3D geometric modeling and 3D video content creation

ABSTRACT

A system, apparatus and method of obtaining data from a 2D image in order to determine the 3D shape of objects appearing in said 2D image, said 2D image having distinguishable epipolar lines, said method comprising: (a) providing a predefined set of types of features, giving rise to feature types, each feature type being distinguishable according to a unique bi-dimensional formation; (b) providing a coded light pattern comprising multiple appearances of said feature types; (c) projecting said coded light pattern on said objects such that the distance between epipolar lines associated with substantially identical features is less than the distance between corresponding locations of two neighboring features; (d) capturing a 2D image of said objects having said projected coded light pattern projected thereupon, said 2D image comprising reflected said feature types; and (e) extracting: (i) said reflected feature types according to the unique bi-dimensional formations; and (ii) locations of said reflected feature types on respective epipolar lines in said 2D image.

RELATIONSHIP TO EXISTING APPLICATIONS

The present application is a Continuation of U.S. patent applicationSer. No. 13/968,961, filed Aug. 16, 2013, which is a Continuation ofU.S. patent application Ser. No. 12/515,715, filed May 20, 2009, whichis the U.S. National Phase of PCT/IL2007/001432, filed Nov. 20, 2007,the contents of which are hereby incorporated by reference.

The present application claims priority from U.S. provisional patent60/860,209, filed on Nov. 21, 2006, the contents of which are herebyincorporated by reference.

The present application also claims priority from U.S. provisionalpatent 60/907,495, filed on Apr. 4, 2007, the contents of which arehereby incorporated by reference.

The present application also claims priority from U.S. provisionalpatent 60/924,206, filed on May 3, 2007, the contents of which arehereby incorporated by reference.

The present application also claims priority from U.S. provisionalpatent 60/929,835, filed on Jul. 13, 2007, the contents of which arehereby incorporated by reference.

The present application also claims priority from U.S. provisionalpatent 60/935,427 filed on Aug. 13, 2007, the contents of which arehereby incorporated by reference.

FIELD AND BACKGROUND OF THE INVENTION

The present invention relates to a system and a method for threedimensional imaging and depth measurement of objects using activetriangulation methods, and, more particularly, but not exclusively, tothree dimensional imaging of both objects at rest and in motion.

Three dimensional sensor systems are used in a wide array ofapplications. These sensor systems determine the shape and or surfacefeatures of an object positioned in a scene of the sensor system's view.In recent years, many methods have been proposed for implementing 3Dmodeling systems that are capable of rapid acquisition of accurate highresolution 3D images of objects for various applications.

The precise configuration of such 3D imaging systems may be varied. Manycurrent triangulation-based systems use an array of at least two or morecameras to determine the depth values by what is known as passive stereocorrespondence. Such a method is dependent on the imaged surfaces beinghighly textured and therefore error prone and non-robust. Furthermore,automatic correspondence algorithms often contain an abundance of errorsin matching between shots from different cameras.

Other methods utilize LIDAR (Light Imaging Detection and Ranging)systems to determine range to and/or other information relating to adistant target. By way of light pulses, the distance to an object isdetermined by measuring the time delay between transmission of the lightpulse and detection of the reflected signal. Such methods, referred toas time-of-flight methods, are generally immune to occlusions typical oftriangulation methods, but the accuracy and resolution are inherentlyinferior to that obtained in triangulation methods.

Active triangulation-based 3D sensor systems and methods typically haveone or more projectors as a light source for projecting onto a surfaceand one or more cameras at a defined, typically rectified, relativeposition from the projector for imaging the lighted surface. The cameraand the projector therefore have different optical paths and thedistance between them is referred to as the baseline. Through knowledgeof the baseline distance, as well as projection and imaging angles,known geometric/triangulation equations are utilized to determinedistance to the imaged object. The main differences among the varioustriangulation methods known in the art lie in the method of projectionas well as the type of light projected, typically structured light, andin the process of image decoding to obtain three dimensional data.

Methods of light projection vary from temporal methods to spatial codedstructured light. Examples in the art of various forms of projectedlight include “laser fans” and line coded light.

Once a 2D image of the object is captured upon which a light source isprojected as described above, image processing software generallyanalyzes the image to extract the three dimensional geometry of theobject and possibly the three dimensional movement of the object throughspace. This is generally done by comparison of features in the capturedimage with previously captured images and/or with known characteristicsand features of the projected light. The implementation of this stepvaries widely among currently known methods, typically a function of themethod used to project light onto the object. Whatever the method used,the outcome of the process is generally a type of disparity/displacementmap of identified features in the captured image. The final step of 3Dspatial location and/or 3D motion capture involves the translation ofthe above mentioned disparity map into depth data, according to wellknown geometric equations, particularly triangulation equations.

The very fact that hundreds of methods and systems exist today hints atthe underlying problem of a lack of a sufficiently effective andreliable method for 3D imaging. Furthermore, most of the systems thatutilize active triangulation methods today are restricted to non dynamicimaging of objects. That is to say, even at high frame rates and shutterspeeds, the imaged object must remain static during image acquisition.For example, a building may be imaged, but not a person riding a bike orcars moving on the street. This limitation on three dimensional imagingis a direct result of the need in most triangulation based 3D imagingsystems to obtain a series of images while changing the characteristicsof the light source over time. For example, many methods utilize anumber of light patterns projected over a time interval, known astemporal coding.

Nonetheless, many methods have been introduced over the years for thethree dimensional imaging of moving objects, most of which are based onthe projection of a single pattern of light on the imaged object, thusenabling reconstruction of the depth measurements from one or moresimultaneous images rather than multiple images over a time interval.These single pattern methods can be broken down into two main classes.The first is assisted stereo methods wherein a single pattern isprojected and a comparison is made between two or more images from twoor more imaging systems to compute depth data.

The second is structured light methods and, in particular, codedstructured light methods. These methods often use only one imagingsystem or camera. Coded structured light methods can further be brokendown into several encoding types. One such type using coded structuredlight is spatial coding, which suffers from a wide range of problems ofprecision and reliability, particularly regarding featureidentification, and other serious performance limitations. As a result,spatial single pattern systems have been implemented commercially onlyin a very limited manner. A further structured coding technique in theart is spectral, or color coding, which requires a color neutral surfaceand usually requires expensive imaging systems.

Therefore, there is an unmet need for, and it would be highly useful tohave, a system and a method that overcomes the above drawbacks.

SUMMARY OF THE INVENTION

According to an embodiment of the present invention, a method ofobtaining data from a 2D (two-dimensional) image in order to determinethe 3D (three-dimensional) shape of objects appearing in the 2D image,said 2D image having distinguishable epipolar lines, may comprise:

providing a predefined set of types of features, giving rise to featuretypes, each feature type being distinguishable according to a uniquebi-dimensional formation,

providing a coded light pattern comprising multiple appearances of saidfeature types;

projecting said coded light pattern on said objects such that thedistance between epipolar lines associated with substantially identicalfeatures is less than the distance between corresponding locations oftwo neighboring features;

capturing a 2D image of said objects having said projected coded lightpattern projected thereupon, said 2D image comprising reflections ofsaid feature types; and

extracting:

-   -   a) said reflected feature types according to the unique        bi-dimensional formations; and    -   b) locations of said reflected feature types on respective        epipolar lines in said 2D image.

According to another embodiment of the present invention, a method ofobtaining data from a 2D image in order to determine the 3D(three-dimensional) shape of objects appearing in said 2D image, said 2Dimage having distinguishable epipolar lines, may comprise:

providing a predefined set of types of features, giving rise to featuretypes, each feature type being distinguishable according to a uniquebi-dimensional formation;

providing a coded light pattern comprising multiple appearances of saidfeature types;

projecting said coded light pattern on said objects at an angle inrelation to said epipolar lines, said angle being in accordance with adetermined limiting distance between said distinguishable epipolarlines;

capturing a 2D image of said objects having said projected coded lightpattern projected thereupon, said 2D image comprising reflections ofsaid feature types; and

extracting:

-   -   a) said reflected feature types according to the unique        bi-dimensional formations; and    -   b) locations of said reflected feature types on respective        epipolar lines in said 2D image.

According to still another embodiment of the present invention, themethod further comprises determining corresponding 3D spatialcoordinates of the objects by means of the locations of the reflectedfeature types on the respective epipolar lines.

According to still another embodiment of the present invention, themethod further comprises providing a 3D point cloud by a compilation ofthe 3D spatial coordinates.

According to still another embodiment of the present invention, themethod further comprises processing the 3D point cloud to form a 3Dsurface.

According to still another embodiment of the present invention, themethod further comprises adding texture data of the objects.

According to a further embodiment of the present invention, the methodfurther comprises providing the coded light pattern as one or more ofthe following:

-   -   a) a repeating periodic pattern; and    -   b) a non-periodic pattern.

According to still a further embodiment of the present invention, themethod further comprises projecting the coded light pattern such thateach of the feature types appears at most once on one or more predefinedsections of the distinguishable epipolar lines.

According to still a further embodiment of the present invention, themethod further comprises providing the entire length of epipolar linesin the 2D image within the predefined sections of the epipolar lines.

According to still a further embodiment of the present invention, themethod further comprises providing the predefined sections as sectionsof the length of the epipolar lines in the 2D image.

According to an embodiment of the present invention, the method furthercomprises projecting the coded light pattern at an angle in relation tothe epipolar lines, in accordance with the parameters of a Sin⁻¹ (P/YC)equation.

According to an embodiment of the present invention, extracting thereflected feature types according to the unique bi-dimensionalformations comprises determining:

-   -   a) elements that comprise said reflected feature types; and    -   b) epipolar distances between said elements of said reflected        feature types.

According to another embodiment of the present invention, the methodfurther comprises providing the elements of the reflected feature typesas bi-dimensional light intensity formations on a pixel area of asensor.

According to still another embodiment of the present invention, themethod further comprises identifying the bi-dimensional light intensityformations by locating critical points of light intensity among adjacentsampled pixel values.

According to still another embodiment of the present invention, themethod further comprises providing the elements as predefined values.

According to still another embodiment of the present invention, themethod further comprises expressing locations of the elements astwo-dimensional coordinates in the 2D image.

According to still another embodiment of the present invention, themethod further comprises providing the epipolar distances between theelements as one or more of the following:

-   -   a) constant distances; and    -   b) varying distances.

According to still another embodiment of the present invention, themethod further comprises providing a spectral encoding within the uniquebi-dimensional formation.

According to still another embodiment of the present invention, themethod further comprises capturing the 2D image at each of successivetime frames.

According to a further embodiment of the present invention, the methodfurther comprises determining, for each of the 2D images, corresponding3D spatial coordinates of the objects by means of the locations of thereflected feature types on the respective epipolar lines for each of the2D images.

According to still a further embodiment of the present invention, themethod further comprises forming a video stream providing 3D motioncapture by means of the 3D spatial coordinates.

According to still a further embodiment of the present invention, themethod further comprises computing the 3D spatial coordinatessubstantially in real-time to provide a substantially real-time videostream of the 3D motion capture.

According to still a further embodiment of the present invention, themethod further comprises generating the bi-dimensional coded lightpattern by superimposing a plurality of values on a predefined pattern,said values derived from one or more of the following:

-   -   a) Debruijn sequences;    -   b) Perfect Maps;    -   c) M-Arrays; and    -   d) Pseudo-random codes.

According to an embodiment of the present invention, an apparatusconfigured to obtain data from a 2D image in order to determine the 3Dshape of objects appearing in said 2D image, said 2D image havingdistinguishable epipolar lines may comprise:

a predefined set of types of features, giving rise to feature types,each feature type being distinguishable according to a uniquebi-dimensional formation;

a coded light pattern comprising multiple appearances of said featuretypes;

a projector configured to project said coded light pattern on saidobjects such that the distance between epipolar lines associated withsubstantially identical features is less than the distance betweencorresponding locations of two neighboring features;

an imaging device configured to capture a 2D image of said objectshaving said projected coded light pattern projected thereupon, said 2Dimage comprising reflected said feature types; and

an image processing device configured to extract:

-   -   a) said reflected feature types according to the unique        bi-dimensional formations; and    -   b) locations of said reflected feature types on respective said        epipolar lines in said 2D image.

According to an embodiment of the present invention, the locations onthe respective epipolar lines of the reflected feature types determinecorresponding 3D spatial coordinates of the objects.

According to another embodiment of the present invention, the projectoris further configured such that the projecting of the coded lightpattern is at an angle in relation to the epipolar lines in accordancewith the parameters of a Sin⁻¹ (P/YC) equation.

According to still another embodiment of the present invention, theapparatus is further configured to move in three-dimensions in relationto the objects during the imaging of said objects.

According to still another embodiment of the present invention, theapparatus is further configured to capture the images of the objectsthat are moving in three dimensions in relation to said apparatus.

According to still another embodiment of the present invention, a methodof obtaining data from a 2D video sequence, comprising two or moreframes, each frame being a 2D image, in order to determine the 3D shapeof moving objects appearing in said 2D video sequence, each frame havingdistinguishable epipolar lines may comprise:

providing a bi-dimensional coded light pattern having a plurality oftypes of features, giving rise to feature types, each feature type beingdistinguishable according to a unique bi-dimensional formation,

projecting said coded light pattern on said moving objects such that thedistance between epipolar lines associated with substantially identicalfeatures is less than the distance between corresponding locations oftwo neighboring features;

capturing said 2D video sequence having said moving objects having saidprojected coded light pattern projected thereupon, said 2D videosequence comprising reflected said feature types, and

extracting:

-   -   a) said reflected feature types according to the unique        bi-dimensional formations; and    -   b) locations of said feature types on respective said epipolar        lines.

According to another embodiment of the present invention, an apparatusconfigured to obtain data from 2D video sequence, comprising two or moreframes, each frame being a 2D image, in order to determine the 3D shapeof moving objects appearing in said 2D video sequence, each frame havingdistinguishable epipolar lines may comprise:

a bi-dimensional coded light pattern having a plurality of types offeatures, giving rise to feature types, each feature type beingdistinguishable according to a unique bi-dimensional formation;

a projector configured to project said coded light pattern on saidmoving objects such that the distance between epipolar lines associatedwith substantially identical features is less than the distance betweencorresponding locations of two neighboring features;

an imaging device configured to capture said 2D video sequence havingsaid moving object having said projected coded light pattern projectedthereupon, said 2D video sequence comprising reflected feature types;and

an image processing device configured to extract:

-   -   a) said reflected feature types according to the unique        bi-dimensional formations; and    -   b) locations of said reflected feature types on respective said        epipolar lines.

According to still another embodiment of the present invention, a methodof obtaining distance data from a scene comprising one or more objectsmay comprise:

projecting a bi-dimensional coded light pattern having a plurality offeature types onto said scene such that each feature of said lightpattern is reflected from a different respective reflection location insaid scene, giving rise to a reflected feature, said feature types beingdistinguishable according to a unique bi-dimensional formation, saidprojecting performed such that the distance between epipolar linesassociated with substantially identical features is less than thedistance between corresponding locations of two neighboring features;

capturing a 2D image of said scene comprising the reflected features;

determining for each said reflected feature appearing in said 2D image:

-   -   a) a respective feature type according to said unique        bi-dimensional formation; and    -   b) a location of said each reflected feature on the respective        epipolar line; and

deriving for said each reflected feature, in accordance with therespective determined feature type and with the respective determinedlocation on said respective epipolar line, a respective distance of arespective associated reflection location.

According to an embodiment of the present invention, the method furthercomprises providing a 3D geometric shape of the one or more objects by acompilation of the respective distances.

According to another embodiment of the present invention, determining ofthe respective feature type and the respective epipolar line compriseseffecting a correspondence between the respective determined featuretype in the 2D image and the location of the same feature type in aprojected pattern.

According to an embodiment of the present invention, a system configuredto obtain scene data from a target scene may comprise:

a pattern-projecting apparatus operative to project a bi-dimensionalcoded light pattern having a plurality of feature types onto said scenesuch that each feature reflects from a respective reflection location,giving rise to a reflected feature, said feature types beingdistinguishable according to a unique bi-dimensional formation, whereinsaid coded light pattern are projected such that the distance betweenepipolar lines associated with substantially identical features is lessthan the distance between corresponding locations of two neighboringfeatures;

an image capture apparatus operative to capture a 2D image of said scenecomprising the reflected features; and

an image processing element operative, for each said reflected featureof said 2D image, to:

-   -   i) determine, for said each reflected feature:        -   A. a feature type according to said unique bi-dimensional            formation; and        -   B. a location of said each reflected feature on the            respective epipolar line; and    -   ii) derive, for each said reflected feature, in accordance with        the respective determined feature type and with the respective        determined location on the respective epipolar line, a        respective distance of a respective associated reflection        location.

According to still another embodiment of the present invention, a methodof determining the 3D shape of imaged objects appearing in at least twoobtained 2D images, said obtained images related to each other bydefined epipolar fields may comprise:

providing a bi-dimensional coded light pattern having a plurality oftypes of features, giving rise to feature types, each feature type beingdistinguishable according to a unique bi-dimensional formation;

projecting said coded light pattern on said imaged objects such that thedistance between epipolar lines associated with substantially identicalfeatures is less than the distance between corresponding locations oftwo neighboring features;

capturing a first 2D image of said imaged objects;

capturing a second 2D image of said imaged objects;

selecting a pixel area PX1 of said first 2D image, the content appearingon PX1 being constrained to appear on a specific epipolar line EPm insaid second 2D image;

finding said content of said pixel area PX1 in said second 2D imagealong said epipolar line EPm; and

determining locations of appearances of said content of said pixel areaPX1 between said first and second 2D images.

According to still another embodiment of the present invention, a methodof determining the 3D shape of imaged objects appearing in at least twoobtained 2D images, said obtained images related to each other bydefined epipolar fields, each epipolar field comprising one or moreepipolar lines may comprise:

providing a bi-dimensional coded light pattern having a plurality oftypes of features, giving rise to feature types, each feature type beingdistinguishable according to a unique bi-dimensional formation;

projecting said coded light pattern on said imaged objects at an anglein relation to at least one of said epipolar fields, said angle being inaccordance with a determined limiting distance between distinguishableepipolar lines;

capturing a first 2D image of said imaged objects;

capturing a second 2D image of said imaged objects;

selecting a pixel area PX1 of said first 2D image, the content appearingon PX1 being constrained to appear on a specific epipolar line EPm insaid second 2D image;

finding said content of said pixel area PX1 in said second 2D imagealong said epipolar line EPm; and

determining locations of appearances of said content of said pixel areaPX1 between said first and second 2D images.

According to an embodiment of the present invention, the method furthercomprises determining corresponding 3D spatial coordinates of the imagedobjects by means of the locations of the appearances of the content ofthe pixel area PX1.

According to an embodiment of the present invention, the method furthercomprises extracting for each of said 2D images, independently,reflected feature types, said feature types being reflected from theimaged objects, according to the unique bi-dimensional formations andthe feature type locations on the respective epipolar lines.

According to still another embodiment of the present invention, anapparatus configured to determine the 3D shape of imaged objectsappearing in two or more obtained 2D images, said two or more 2D imagesbeing obtained from at least two imaging devices, said two or more 2Dimages being related to each other by defined epipolar fields maycomprise:

a bi-dimensional coded light pattern having a plurality of types offeatures, giving rise to feature types, each feature type beingdistinguishable according to a unique bi-dimensional formation;

a projector configured to project said coded light pattern on saidimaged objects such that the distance between epipolar lines associatedwith substantially identical features is less than the distance betweencorresponding locations of two neighboring features;

a first imaging device configured to capture a first 2D image of saidimaged objects;

a second imaging device configured to capture a second 2D image of saidimaged objects;

an image processing configured to:

-   -   a) select a pixel area PX1 of said first 2D image, the content        appearing on said pixel area PX1 being constrained to appear on        a specific epipolar line EPm in the second 2D image;    -   b) find said content of said pixel area PX1 in said second 2D        image along said epipolar line EPm; and    -   c) determine locations of appearances of said content of said        pixel area PX1 between said first and second 2D images.

According to an embodiment of the present invention, the locations ofthe appearances of the content of the pixel area PX1 determinecorresponding 3D spatial coordinates of the imaged objects.

According to another embodiment of the present invention, the imageprocessing device is further configured to extract in each image,independently, the reflected feature types, said feature types beingreflected from the imaged objects, according to the uniquebi-dimensional formations and the feature type locations on therespective epipolar lines.

According to still another embodiment of the present invention, a methodof obtaining data from a 2D image in order to determine the 3D shape ofobjects appearing in said 2D image, said 2D image having distinguishableepipolar lines may comprise:

providing a bi-dimensional coded light pattern having a plurality offeature types, each of said feature types being distinguishableaccording to a unique bi-dimensional formation;

providing the inverse of said bi-dimensional coded light pattern, givingrise to an inverse coded light pattern;

projecting said coded light pattern and said inverse coded light patternon said objects;

capturing:

-   -   i) a first 2D image of said objects, having said projected coded        light pattern projected thereupon, said first 2D image        comprising reflected said feature types; and    -   ii) a second 2D image of said objects, having said inverse coded        light pattern projected thereupon, said second 2D image        comprising reflected said feature types;

obtaining a resultant image from the subtraction of said second 2D imagefrom said first 2D image; and

extracting said reflected feature types according to the uniquebi-dimensional formations and the feature type locations on respectiveepipolar lines in said resultant image.

According to an embodiment of the present invention, the method furthercomprises projecting the light patterns by one or more of the following:

-   -   a) temporally;    -   b) with differing spectral values of light; and    -   c) with differing polarization.

According to another embodiment of the present invention, the methodfurther comprises capturing the first 2D image and second 2D imagestemporally, giving rise to temporal imaging.

According to still another embodiment of the present invention, themethod further comprises carrying out the temporal imaging overnon-uniform time intervals.

According to still another embodiment of the present invention, themethod further comprises capturing the first and second 2D imagessimultaneously by using spectral differentiation.

According to a further embodiment of the present invention, the methodfurther comprises carrying out the extraction of the feature types inthe resultant image by determining intensity values of sample points.

According to still another embodiment of the present invention, anapparatus configured to obtain data from a 2D image in order todetermine the 3D shape of objects appearing in said 2D image, said 2Dimage having distinguishable epipolar lines may comprise:

a predefined set of feature types, each feature type beingdistinguishable according to a unique bi-dimensional formation;

a bi-dimensional coded light pattern comprising multiple appearances ofthe feature types;

an inverse bi-dimensional coded light pattern comprising multipleappearances of the inversed feature types;

a projector configured to project said bi-dimensional coded lightpattern and said inverse bi-dimensional light pattern on said objects;

a first imaging device configured to capture a first 2D image of saidobjects, having said projected coded light pattern projected thereupon,said first 2D image comprising reflections of said feature types;

a second imaging device configured to capture a second 2D image of saidobjects, having said inverse coded light pattern projected thereupon,said second 2D image comprising reflections of said feature types; and

an image processing device configured to:

-   -   i) obtain a resultant image from the subtraction of said second        2D image from said first 2D image; and    -   ii) extract said reflected feature types according to the unique        bi-dimensional formations and the feature type locations on        respective said epipolar lines in said resultant image.

According to a further embodiment of the present invention, a method ofobtaining texture data from two 2D images, each of said imagescontaining a reflected code used to obtain depth data of imaged objectsindependently for each image may comprise:

providing a bi-dimensional coded light pattern having a plurality offeature types, each feature type being distinguishable according to aunique bi-dimensional formation, providing the inverse of saidbi-dimensional coded light pattern, giving rise to an inverse codedlight pattern;

projecting said coded light pattern and said inverse coded light patternon said objects;

capturing:

-   -   i) a first 2D image of said objects, having said projected coded        light pattern projected thereupon, said first 2D image        comprising reflected said feature types, ii) a second 2D image        of said objects, having said inverse coded light pattern        projected thereupon, said second 2D image comprising reflected        said feature types, obtaining a resultant image from the        addition of said second 2D image with said first 2D image, said        resultant image providing texture information of said imaged        objects.

According to still a further embodiment of the present invention, amethod of obtaining data from a 2D image in order to determine the 3Dshape of objects appearing in said 2D image, said 2D image havingdistinguishable epipolar lines may comprise:

providing a bi-dimensional coded light pattern having a plurality offeature types, each feature type being distinguishable according to aunique bi-dimensional formation;

providing the inverse of said bi-dimensional coded light pattern, givingrise to an inverse coded light pattern;

projecting said coded light pattern and said inverse coded light patternon said objects;

capturing:

-   -   i) a first 2D image of said objects, having said projected coded        light pattern projected thereupon, said first 2D image        comprising reflected said feature types; and    -   ii) a second 2D image of said objects, having said inverse coded        light pattern projected thereupon, said second 2D image        comprising reflected said feature types;

obtaining a resultant image from the absolute value of the subtractionof said second 2D image from said first 2D image, said resultant imagecomprising outlines of said reflected feature types; and

extracting said outlines and outline locations on respective saidepipolar lines in said resultant image.

According to an embodiment of the present invention, the method furthercomprises determining corresponding 3D spatial coordinates of theobjects by means of the outline locations on the respective epipolarlines.

According to still a further embodiment of the present invention, amethod of obtaining data from a 2D image in order to determine the 3Dshape of objects appearing in said 2D image, said 2D image havingdistinguishable epipolar lines may comprise:

providing a first bi-dimensional coded light pattern having a pluralityof feature types, each feature type being distinguishable according to aunique bi-dimensional formation and each feature type comprising aplurality of elements having varying light intensity, wherein saidplurality of elements comprises non-maximum and non-minimum elements andcomprises one or more of the following:

-   -   a) at least one maximum element;    -   b) at least one minimum element; and    -   c) at least one saddle element;

providing a second bi-dimensional coded light pattern comprising saidfirst bi-dimensional coded light pattern having the one or more maximumand/or minimum inverted elements;

projecting said first bi-dimensional coded light pattern and said secondbi-dimensional coded light pattern on said objects;

capturing:

-   -   i) a first 2D image of said objects, having said first projected        bi-dimensional coded light pattern projected thereupon, said        first 2D image comprising reflected said feature types; and    -   ii) a second 2D image of said objects, having said second        bi-dimensional coded light pattern projected thereupon, said        second 2D image comprising reflected said feature types;

obtaining a resultant image from the subtraction of said second 2D imagefrom said first 2D image; and

extracting from said resultant image one or more of the following: atleast one maximum element, at least one minimum element and at least onesaddle element, and extracting their locations respective to saidepipolar lines in said resultant image.

According to an embodiment of the present invention, the method furthercomprises determining the corresponding feature types by determiningtheir locations and/or by determining values of the one or more of thefollowing: at least one maximum element, at least one minimum element,at least one non-maximum element, at least one non-minimum element andat least one saddle element.

According to another embodiment of the present invention, the methodfurther comprises determining corresponding 3D spatial coordinates ofthe objects by determining locations of the one or more of thefollowing: at least one maximum element, at least one minimum element,and at least one saddle element and/or by determining locations of thenon-maximum and non-minimum elements on the respective epipolar lines.

According to still a further embodiment of the present invention, amethod of obtaining data from a 2D image in order to determine the 3Dshape of objects appearing in said 2D image, said 2D image havingdistinguishable epipolar lines may comprise:

providing a first bi-dimensional coded light pattern having a pluralityof feature types, each feature type being distinguishable according to aunique bi-dimensional formation and each feature type comprising aplurality of elements having varying light intensity, wherein saidplurality of elements comprising non-maximum and non-minimum elementscomprising one or more of the following:

-   -   a) at least one maximum element;    -   b) at least one minimum element; and    -   c) at least one saddle element;

providing a second bi-dimensional coded light pattern comprising saidfirst bi-dimensional coded light pattern having the one or more maximumand/or minimum inverted elements;

projecting said first bi-dimensional coded light pattern and said secondbi-dimensional coded light pattern on said objects;

capturing:

-   -   i) a first 2D image of said objects, having said first projected        coded light pattern projected thereupon, said first 2D image        comprising reflections of said feature types; and    -   ii) a second 2D image of said objects, having said second coded        light pattern projected thereupon, said second 2D image        comprising reflected said feature types;

obtaining a resultant image from the addition of said second 2D imagewith said first 2D image; and

extracting, from said resultant image, one or more of said non-maximumand non-minimum elements and locations of said non-maximum andnon-minimum elements on one or more respective epipolar lines in saidresultant image.

According to still a further embodiment of the present invention, amethod of obtaining data from a 2D image in order to determine the 3Dshape of objects appearing in said 2D image, said 2D image havingdistinguishable epipolar lines may comprise:

providing a first bi-dimensional coded light pattern having a pluralityof feature types, each feature type being distinguishable according to aunique bi-dimensional formation and each feature type comprising aplurality of elements having varying light intensity, wherein saidplurality of elements comprises non-maximum and non-minimum elements andcomprises one or more of the following:

-   -   a) at least one maximum element;    -   b) at least one minimum element; and    -   c) at least one saddle element;

providing a second bi-dimensional coded light pattern comprising saidfirst bi-dimensional coded light pattern having the one or more maximumand/or minimum inverted elements;

projecting said first bi-dimensional coded light pattern and said secondbi-dimensional coded light pattern on said objects;

capturing:

-   -   i) a first 2D image of said objects, having said first projected        bi-dimensional coded light pattern projected thereupon, said        first 2D image comprising reflected said feature types; and    -   ii) a second 2D image of said objects, having said second        bi-dimensional coded light pattern projected thereupon, said        second 2D image comprising reflected said feature types;

obtaining a resultant image from the absolute value of the subtractionof said second 2D image from said first 2D image, said resultant imagecomprising borderlines of said reflected feature types; and

extracting said borderlines and borderline locations on respective saidepipolar lines in said resultant image.

According to an embodiment of the present invention, the method furthercomprises determining corresponding 3D spatial coordinates of theobjects by means of the borderline locations on the respective epipolarlines.

According to still a further embodiment of the present invention, amethod of obtaining data from a 2D image in order to determine the 3Dshape of objects appearing in said 2D image, said 2D image havingdistinguishable epipolar lines may comprise:

providing a first bi-dimensional coded light pattern having a pluralityof feature types, each feature type being distinguishable according to aunique bi-dimensional formation;

providing a second bi-dimensional coded light pattern, different fromsaid first pattern, having a plurality of feature types, each featuretype being distinguishable according to a unique bi-dimensionalformation;

projecting said first coded light pattern and said second coded lightpattern on said objects;

capturing:

-   -   i) a first 2D image of said objects having said first projected        coded light pattern projected thereupon, said first 2D image        comprising reflected said feature types; and    -   ii) a second 2D image of said objects having said second coded        light pattern projected thereupon, said second 2D image        comprising reflected said feature types;

extracting, in each image independently, said reflected feature typesaccording to the unique bi-dimensional formations and the feature typelocations on respective epipolar lines; and comparing regions ofepipolar lines in said second 2D image to similar regions along sameepipolar lines in said first 2D image to verify feature identities insaid first 2D image.

According to still a further embodiment of the present invention, amethod of obtaining data from a 2D image in order to determine the 3Dshape of objects appearing in said 2D image, said 2D image havingdistinguishable epipolar lines may comprise:

providing a bi-dimensional coded light pattern having a plurality offeature types, each feature type being distinguishable according to aunique bi-dimensional formation;

projecting said bi-dimensional coded light pattern on said objects;

capturing:

-   -   i) a first 2D image of said objects, having said projected coded        light pattern projected thereupon, said first 2D image        comprising reflections of said feature types; and    -   ii) a second 2D image of said objects, having ambient light        projected thereupon;

obtaining a resultant image from the subtraction of said second 2D imagefrom said first 2D image; and

extracting said reflected feature types and locations of said reflectedfeature types on respective said epipolar lines in said resultant image.

According to still a further embodiment of the present invention, amethod of obtaining data from a 2D image in order to determine the 3Dshape of objects appearing in said 2D image, said 2D image havingdistinguishable epipolar lines may comprise:

providing a bi-dimensional coded light pattern having a plurality offeature types, each feature type being distinguishable according to aunique bi-dimensional formation;

projecting said coded light pattern on said objects;

capturing:

-   -   i) a first 2D image of said objects, having said projected coded        light pattern projected thereupon, said first 2D image        comprising reflected said feature types; and        -   ii) a second 2D image of said objects, having uniform light            projected thereupon;

obtaining a resultant image from the division of said first 2D image bysaid second 2D image; and

extracting said reflected feature types and the locations of saidreflected feature types on respective epipolar lines in said resultantimage.

Unless otherwise defined, all technical and scientific terms used hereinhave the same meaning as commonly understood by one of ordinary skill inthe art to which this invention belongs. The materials, methods, andexamples provided herein are illustrative only and not intended to belimiting.

Implementation of the method and system of the present inventioninvolves performing, or completing, certain selected tasks or stepsmanually, automatically, or a combination thereof. Moreover, accordingto actual instrumentation and equipment of preferred embodiments of themethod and system of the present invention, several selected steps couldbe implemented by hardware or by software on any operating system of anyfirmware or a combination thereof. For example, as hardware, selectedsteps of the invention could be implemented as a chip or a circuit. Assoftware, selected steps of the invention could be implemented as aplurality of software instructions being executed by a computer usingany suitable operating system. In any case, selected stages of themethod and system of the invention could be described as being performedby a data processor, such as a computing platform for executing aplurality of instructions.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is herein described, by way of example only, withreference to the accompanying drawings. With specific reference now tothe drawings in detail, it is stressed that the particulars shown are byway of example and for purposes of illustrative discussion of thepreferred embodiments of the present invention only and are presented inorder to provide what is believed to be the most useful and readilyunderstood description of the principles and conceptual aspects of theinvention. In this regard, no attempt is made to show structural detailsof the invention in more detail than is necessary for a fundamentalunderstanding of the invention, the description taken with the drawingsmaking apparent to those skilled in the art how the several forms of theinvention may be embodied in practice.

In the drawings:

FIG. 1A-1G are simplified diagrams illustrating one embodiment of thepresent invention showing how bi-dimensional light pattern imaging isused together with the principles of epipolar geometry.

FIG. 2 is a simplified flow chart showing the steps in the process ofthree dimensional image and/or motion-image capture of the presentembodiments.

FIG. 3 illustrates an exemplary spatially-periodic 2D pattern P1projected to obtain epipolar separation and corresponding image ‘PI ofthe reflected pattern.

FIG. 4 is a simplified flow chart and diagram illustrating how thecorresponding 3D spatial locations of identified features in thecaptured 2D image are derived.

FIG. 5 shows further exemplary point clouds derived through the methodsof embodiments of the present invention.

FIG. 6A is a simplified schematic representation of a reflected patternafter being projected in accordance with the epipolar separationprinciple.

FIG. 6B again shows a simplified schematic representation of a reflectedpattern, but now includes multiple appearances of the same feature on agiven epipolar line in the image.

FIG. 7 is a simplified illustration of a captured image of a preferredlight pattern projected in accordance with a particular embodiment ofthe present invention.

FIG. 8 is a simplified schematic representation of a pattern nowprojected at a smaller rotation angle with respect to the direction ofthe epipolar field.

FIG. 9 is a simplified schematic diagram of a non ordered and nonperiodic bi dimensional spatial pattern that may be used in certainembodiments.

FIG. 10 shows illustrations of exemplary feature types that comprisepreferred encoded pattern P1.

FIG. 11 is an illustration of the exemplary pattern P1 as seen an imageI′_(P1) after being reflected from an imaged cube.

FIGS. 12A and 12B illustrate the construction process and the encodingscheme inherent in pattern P1.

FIG. 12C contains simplified illustrations of the sample points inpattern P1 after being reflected from an imaged object and viewed aspart of a captured image in an imaging apparatus.

FIG. 13A illustrates a preferred projection and imaging method usingpattern P1 that enable epipolar separation techniques.

FIG. 13B1 illustrates a preferred pattern P2 that provides an increasednumber of sample points.

FIG. 13B2 illustrate the construction process and the encoding schemeinherent in pattern P2.

FIG. 13C illustrates two preferred patterns, T1 and T2, for a temporalencoding embodiment.

FIG. 13D illustrates a close-up of preferred pattern T2 for a temporalencoding embodiment.

FIG. 14 shows exemplary characters or feature types along with imageI_(P1) of exemplary pattern P1.

FIG. 15 is an illustration of a preferred dual projection and dualimaging embodiment of the present invention involving P1 and −P1.

FIG. 16 is an illustration of the addition of the two images, P₁ and−P₁.

FIG. 17 is a preferred embodiment of a dual projection and dual imagingmethod of the present invention.

FIG. 18 shows the addition of the two imaged patterns of FIG. 17.

FIG. 19 shows an illustration of the resultant image obtained from theabsolute value of the subtraction of image −P₁ from image P₁.

FIG. 20 is an illustration of the resultant image obtained from theabsolute value of the subtraction of image −C from image P₁.

FIG. 21 is an illustration of a particular embodiment of dual patternprojection and dual pattern imaging.

FIG. 22 is an illustration of a particular embodiment of dual projectionusing uniform light.

FIG. 23 is a simplified illustration showing how an epipolar fieldbetween two images is related.

FIG. 24 is a simplified illustration showing how separate epipolarfields exist for respective imaging apparatuses and a projector in aparticular embodiment.

FIG. 25 is a simplified flow chart of a particular embodiment of thepresent invention showing the steps in the process of generating a 3Dmap of an urban region.

FIG. 26 is simplified flow chart of a particular embodiment of thepresent invention showing the steps in the process of 3D imaging of ananatomical part.

FIG. 27 is a typical flow chart of a particular embodiment of thepresent invention showing the steps in the process of determiningdistances of obstacles in the path of a reversing motor vehicle.

DESCRIPTION OF THE PREFERRED EMBODIMENTS

The present embodiments provide an apparatus and a method for threedimensional imaging of static and moving objects. In particular, it ispossible to determine three-dimensional spatial shape data of a givenobject(s) by (a) projecting an encoded bi-dimensional light pattern onthe object(s); and (b) analyzing a 2D image of the reflectedbi-dimensional light pattern utilizing the optical constraints andprinciples associated with epipolar geometry and triangulation.

The principles and operation of an apparatus and method according to thepresent invention may be better understood with reference to thedrawings and accompanying description.

Before explaining at least one embodiment of the invention in detail, itis to be understood that the invention is not limited in its applicationto the details of construction and the arrangement of the components setforth in the following description or illustrated in the drawings. Theinvention is capable of other embodiments or of being practiced orcarried out in various ways. Also, it is to be understood that thephraseology and terminology employed herein is for the purpose ofdescription and should not be regarded as limiting.

FIGS. 1A-1G: Determining a 3D Depth Map of an Object or Objects

Reference is now made to FIG. 1A-1D, which are simplified diagramsillustrating one embodiment of the present invention. In the particularexample of FIGS. 1A-1D, system 10 comprises projector 12 and imagingapparatus 14 found at respective locations (x₁ ^(p), y₁ ^(p), z₁ ^(p))and (x₁ ^(i), z₁ ^(i)). An encoded bi-dimensional light pattern 16,comprising a predefined array of a finite set of identifiable featuretypes, is projected by the projector 12 onto surfaces in a scenecontaining imaged objects 18 a, 18 b, and 18 c. Exemplary features inthe projected pattern are denoted as 20 a, 20 b, 20 d, and 20 f. In thisparticular example, the projected pattern takes the form of an array ofmonochromatic light beams of varying intensity, wherein combinations ofadjacent light beams comprise encoded features or letters havingbi-dimensional spatial formations. These features intersect the imagedobjects' surfaces at various reflection points 19 a, 19 b, 19 d, and 19f and are then observed in 2D image 22 and captured on sensor 24 ofimaging apparatus 14. The projected features 20 a, 20 b, 20 d, and 20 fthus appear as reflected features 28 a, 28 b, 28 d, and 28 f in thecaptured image 22. Imaging apparatus 14 may be, for example, a CCD orCMOS digital video camera, or any other type of array of photo-detectorelements.

The relative position 30 between projector 12 and sensor 24 is fixed,thus imparting a geometric constraint to system 10. This constraintrestricts the location of any given reflected feature 28 a, 28 b, 28 d,and 28 f in the captured image 22 to a unique set of points in the imagecalled an epipolar line. Epipolar lines β, φ, and ρ are shown in thefigure. That is to say, for a system 10 with a given light pattern andfixed relative position between projector and sensor, for any object orgroup of objects upon which the pattern is projected, each feature ofthe projected pattern is always observed on a particular, singularepipolar line in the captured image 22. Moreover, as will be explainedbelow, the above holds true whether the imaged objects are fixed or inrelative motion to the projector/imaging apparatus. The relationshipbetween each feature in the pattern and a constant epipolar line is thusirrespective of the reflection point in 3D space from which the featureis reflected.

The reflection points of the exemplary projected features are denoted as19 a, 19 b, 19 d, and 19 f and have associated respective reflectiondistances D_(A), D_(B), D_(D), and D_(F). The reflection distance is thedistance from the sensor 24 of the imaging apparatus to the reflectionpoint location in 3D space on the imaged object 18 a, 18 b, 18 c uponwhich a given feature is reflected. Reflection distance is sometimesreferred to as depth.

Thus, as illustrated in FIGS. 1A-1D, each reflected feature 28 a, 28 b,28 d, and 28 f found in image 22 is constrained to a respective specificepipolar line, independent of the point in space from which the featurewas reflected. Feature 28 a is thus constrained to epipolar line φindependent of the 3D location of 19 a, or alternatively independent ofreflection distance D_(A). Features 28 b and 28 d are two reflectedfeatures from the pattern that share the same epipolar line β. Features28 b and 28 d are always found on this epipolar line irrespective ofreflection distances D_(B) and D_(D). Similarly, feature 28 f is alwaysreflected onto epipolar line ρ, independent of the reflection distanceD_(F).

However, the reflection distance, or alternatively the point in 3D spacefrom which the feature is reflected, does indeed determine where alongthat epipolar line the feature appears. Thus, the reflection distancedoes not affect the particular epipolar line upon which a givenreflected feature appears, but rather only the precise location alongthat line.

To illustrate the change in position of a feature along the epipolarline as a function of reflection distance, we now turn to FIGS. 1E-1G.In each of FIGS. 1E, 1F, and 1G, the projector and imaging apparatus,although always at a fixed relative distance to one another, move inrelation to the object 18 a. In FIG. 1E, projector 12 and imagingapparatus 14 are at locations (x₁ ^(p), y₁ ^(p), z₁ ^(p)) and (x₁ ^(i),y₁ ^(i), z₁ ^(i)) respectively. The reflection distance of reflectedfeature 28 a (projected feature 20 a) on epipolar line φ is seen asD_(A), after being reflected from reflection point 19 a on object 18 a.In FIG. 1F, the projector and imaging apparatus move to respectivepoints in space (x₂ ^(p), y₂ ^(p), z₂ ^(p)) and (x₂ ^(i), y₂ ^(i), z₂^(i)). The reflection distance of reflected feature 28 a is now D_(A)′after being reflected from a different reflection point 19 a′ on object18 a. As a result, feature 28 a is now reflected onto a lower part ofepipolar line φ. In FIG. 1G, the projector and imaging apparatus move toa third set of respective coordinates in space (x₃ ^(p), y₃ ^(p), z₃^(p)) and (x₃ ^(i), y₃ ^(i), z₃ ^(i)). The reflection distance ofreflected feature 28 a is now D_(A)″ after being reflected from a stillthird reflection point 19 a″ on object 18 a. As a result, feature 28 ais now reflected onto a higher part of epipolar line φ. However, nomatter what the reflection distance, reflected feature 28 a (projectedfeature 20 a) must always appear on epipolar line φ and only on φ. Forpurposes of clarity, each feature is associated with only one epipolarline in the present embodiment. It is understood that, as featurescomprise spatial formations of light intensities in the captured image,elements of each feature may lie on separately distinguishable epipolarlines.

It is understood that any relative motion between the system 10 and animaged object causes the reflection distance of a particular feature tochange. This relative motion may result from motion of the imagedobject, from motion of the projector/imaging system, or from motion ofboth the imaged object and projector/imaging system. Any change in thereflection distance causes a change in the precise location of a givenreflected feature along that feature's associated epipolar line.However, the particular epipolar line upon which that feature appearsremains constant.

Therefore, we can conclude that the principles of epipolar geometrydictate a mapping between a set of 2D coordinates in the image and threedimensional coordinates in space that are viewed by the imagingapparatus. Again, the precise point at which a given captured featureappears along a particular epipolar line is a function of the feature'sreflection distance, or alternatively stated, the point in 3D space fromwhich the feature is reflected.

For each image, or frame, each reflected feature is identified accordingto feature type and the reflected feature's location is determined alongthe feature's associated epipolar line in the image, preferably by animage processing device 36. The precise position of an identifiedfeature along the feature's epipolar line is then corresponded to thatfeature's position in the original projected pattern. Thiscorrespondence of features between the image and projected patternallows for triangulation based computations to determine the 3D spatialcoordinate from which the feature was reflected. Pre-computedtriangulation tables may be utilized to determine the three dimensionalspatial location of the point on the object from which the feature isreflected. In some embodiments, these triangulation based calculationsare carried out by the image processing device 36.

This process may be repeated for a plurality of features of the 2Dimage, where each feature is reflected off of a different respectivelocation on the surface of the 3D imaged object(s). For any given imageframe, each such identified feature in the captured 2D image leads to athree dimensional spatial location; the compilation of all such spatiallocations comprising a point cloud of locations in 3D space. This 3Dpoint cloud gives a three dimensional mapping of the imaged object(s).Further processing of the point cloud may yield a 3D mesh, whichessentially fuses the points of the 3D cloud into 3D surfaces. This meshmay also be given graphics texture based on additional texture captureof the objects in the scene. For 3D mapping of objects in motion, theabove process described for a single image is carried out over a seriesof images leading to 3D video.

FIG. 2: Simplified Flow Chart of 3D Image Capture

Reference is made to FIG. 2, which is a simplified flow chart showingthe steps in the process of three dimensional image and/or motion-imagecapture of the present embodiments. Each step of the flow chart isdiscussed in further detail in the following figures. The flow chart,together with diagrams 1A-1G, gives the reader a simplified and overallintuitive understanding of the three dimensional imaging processdescribed herein.

Step 70 is the provision of a predefined coded light pattern. This codedlight pattern, as exemplified in FIGS. 1A-1D and denoted there as 16, isan array of a finite set of feature types in the form of spatialformations of varying intensity light beams. Preferred embodiments ofpatterns and their characteristics are presented in the figures anddiscussed below. Step 72 is the projection of that pattern on anobject(s). Several preferred projection methods are presented in thefigures and discussed below as well. In step 74, a 2D image is capturedthat contains features reflected off of the object(s) upon which thepattern was projected. The image is analyzed to identify features andtheir locations along respective epipolar lines, step 76. The locationsof the features along their epipolar lines are then associated with 3Dcoordinates in space from which the features were reflected, step 78.This process of correlating between feature locations along epipolarlines and 3D spatial coordinates, carried out through triangulationtechniques, is discussed below. For each identified feature in the 2Dimage, a corresponding 3D coordinate is thus derived indicating thepoint in space at which that feature reflected off of an imaged object.Through a compilation of all such 3D coordinates, a 3D point cloud isderived that gives a three dimensional mapping of the imaged object(s),step 80. Further processing of the point cloud may yield a 3D mesh whichessentially fuses the points of the 3D cloud into 3D surfaces. This meshmay also be given graphics texture based on additional texture captureof the objects in the scene.

In the case where objects or the camera are in motion, steps 74-80 maybe continuously repeated to obtain 3D motion capture. In such a case of3D motion capture, a series of 2D captured images of the reflected lightpattern off of the moving object(s) comprises frames in a videosequence. This 2D video sequence can be processed frame by frame in themanner discussed in the flow chart to derive the 3D spatial coordinatesfor each frame. The result is a series of point clouds, one for eachframe of the video sequence, that together comprise a dynamic 3D mappingover time.

FIG. 3: Pattern Projection and Epipolar Separation

FIG. 3 provides an exemplary spatially-periodic 2D pattern P1 which maybe projected onto the object. The 2D pattern includes a plurality ofdifferent feature types 20, which appear within the pattern at variouslocations. For clarity, in the present figure, each unique feature typeis arbitrarily assigned an alphabetic character. As will be explained infurther detail below in FIG. 10, each pattern feature is composed of acodified spatial formation of black and white points. Features arerepeated in a cyclical fashion in the code. Specifically, identicalfeature types repeat themselves every other row in the verticaldirection of the pattern and every sixteen columns in the horizontaldirection of the pattern. The black and white points of the patternfeatures correspond to projections of either a high or low illuminationintensity of monochromatic light on an imaged object. Each character inthe code is thus a bidimensional spatial formation of projected lightintensities. In the pattern of the present embodiment, P1, theformations are continuous. Other patterns having non-continuousformations may also be implemented.

After reflection off of the 3D objects, subsection 48 of the projectedperiodic pattern P1 generates image I_(P1), containing the reflectedpattern. Epipolar lines 52, 54, and 56 are shown. Thus image I_(P1) is asimplified illustration of the reflected pattern being viewed through animaging apparatus upon being reflected off of an imaged object. Forsimplicity, only the reflected pattern is shown and not any reflectedimaged objects appearing in I_(P1) as well. In this particularembodiment, the projector and imaging apparatus are positionedvertically from one another, thus causing the epipolar lines in thecaptured image to be in a substantially vertical direction. In anotherpossible embodiment, the projector and imaging apparatus may bepositioned horizontally from each other, in which case the epipolarlines in the captured image would be substantially horizontal.

As seen and mentioned above, any given feature type repeats itselfcyclically many times in the vertical direction of the pattern, and,were the pattern to be projected without rotation, many features of thesame type would be observed in the captured image on the same verticalepipolar line. However, through slight rotation of the projectedpattern, features of the same type are reflected onto separate epipolarlines in image I_(P1). For example, features 58 and 60, both of type(A), are now reflected onto two adjacent but distinguishably separateepipolar lines 52 and 54. As will be explained further below, theseparation of all identical pattern feature types onto separate epipolarlines in the captured image, called epipolar separation herein, enablesidentification of each feature without ambiguity. To understand why thisis so, a brief discussion of system calibration follows to understandthe relationship between features in the original projected pattern andepipolar lines in the captured image.

As explained above, given a particular pattern such as that described inthe current embodiment, and assuming the geometric constraint of a fixedrelative position between the projector and imaging apparatus, any givenpattern feature appears on a constant pre-determined epipolar line in animage containing the reflected pattern. This pre-determined relationshipbetween each feature in the projected pattern and an associated epipolarline in an image of the reflected pattern may be determined in manyways, including brute force. The process of determining thisrelationship may be referred to as the epipolar field calibrationprocess, and is carried out a single time for a given pattern and givenconfiguration of the imaging device and projector. The result of such amapping is an epipolar-field table as seen below.

Feature Type and Location in Pattern Epipolar Line In Captured linage(A₁) at X₁, Y₁ E₁ (A₂) at X₂, Y₂ E₂ . . . . . . (A_(N)) at X₃, Y₃ E_(N)(B₁) at X₄, Y₄ E₁ (B₂) at X₅, Y₅ E₂ . . . . . . (B_(M)) at X₆, Y₆ E_(N). . . . . .

As seen from the table, each occurrence of a feature type in the patternis associated with a different epipolar line: (A₁) with E₁, (A₂) withE₂, and so on for as many (A) type appearances in the projected patternthat appear in the image. Of course features of different types, such as(A₁) and (B₁), can fall on the same epipolar line, such as E₁. Thus, allepipolar line locations and the specific features in the originalpattern that fall on these epipolar lines are known from the calibrationprocess.

As mentioned, any given image containing the reflected pattern, containsfeature types of a set of a certain size. In the pattern of the currentembodiment this set contains 32 types. The decoding engine used todecode the captured image decodes each feature according to feature typeand location coordinate in the image, including epipolar line. Thisdecoded image feature is then corresponded with a particular feature inthe original projected pattern. For instance, if a feature type (A) isfound on epipolar line E₂, then from the epipolar field calibrationtable we know that this image feature uniquely corresponds to feature(A₂) in the original pattern, and not any other feature in the pattern.Again, were the cyclical pattern to be projected without rotation, manyfeatures of the same type would be observed in the captured image on thesame epipolar line. Since reflected features are identified based onfeature type, there would be no way of corresponding on a one-to-onebasis between identical image feature types and their correspondingfeatures in the original pattern. In other words, there would be noindication of which identified (A) feature type in the image correspondsto which (A) type feature in the projected pattern.

By limiting each feature type to a singular appearance on a particularepipolar line in the captured image, however, the association of thatparticular image feature and the feature's appearance in the pattern canbe made. Therefore each epipolar line has at most one feature of a givenfeature type. So each (A) type feature in the pattern P1 falls on adifferent epipolar line in the image, each (B) type feature falls on anindependent epipolar line, and so on for all pattern features.Similarly, each epipolar line preferably has multiple features so longas they are of distinguishable types. For instance, with reference toFIGS. 1A-1D, it is noted that features 28 b and 28 d are both found onthe same epipolar line β, yet may be compared to respective patternappearances since they are of different, distinguishable feature types.

The unambiguous matching between a particular image feature and thatfeature's appearance in the pattern leads to a correct triangulationthat determines a precise 3D location in space from which the feature isreflected.

It is appreciated that the present embodiments utilize a rectifiedimaging system, meaning that the plane of the imaging apparatus and theplane of the projector lie on the same spatial plane. As such, only avertical or horizontal shift exists between the projector and theimaging apparatus. In such a rectified system, the epipolar lines in thecaptured image together comprise a unidirectional epipolar field. Sincethe epipolar lines in the image are thus parallel to each other, therotation of the repeating pattern places repeating feature types ontoseparate epipolar lines.

So in summation, as a result of the internal structure of the code andthe projection approach, each epipolar line has at most one feature of agiven feature type. Again, although the pattern is cyclical andtherefore feature types appear many times in the pattern, the abovedescribed placement of same feature types onto separate epipolar linesmeans there is no confusion between multiple appearances of that featuretype in the image. For each image in a stream of images, the location ofeach feature determines a new set of 3D spatial coordinates that providea 3D map of the currently imaged object location.

With reference to FIGS. 1A-1G, image processing device 36 preferablycontains at least the image processing engine used to identify (1)features and their associated epipolar lines in the captured 2D imageand (2) feature locations upon their epipolar lines. The imageprocessing device typically also has a database containing thecoordinates for all features in the projected pattern. The processingdevice matches each identified feature in the image to that feature'scorresponding feature in the pattern. Triangulation-based computationsthen are used to assign three dimensional coordinates in space to eachidentified feature to derive a 3D point cloud of the imaged object.These triangulations-based computations are preferably stored in a tablein memory to aid in fast processing of 3D spatial coordinates. Furtherprocessing of the point cloud may be optionally desired. Such processingmay include determining a 3D surface or mesh from the 3D point cloud fora richer, life-like 3D image/video. Texture data may be added as well.Such processing of the point cloud may utilize known processing enginesfor three dimensional point cloud processing.

As such, through novel techniques described herein combining the aboveprinciples of epipolar geometry together with structured lightprojection and encoding/decoding schemes, the present embodimentsprovide for high resolution, high accuracy 3D imaging, such as thatprescribed for HDTV, 4CIF, mega-pixel imaging and similar applications.In preferred embodiments, for short distances, accuracy is obtained ofless than one millimeter. That is to say, the triangulation error is +−1millimeter.

As stated, this system and method may be preferably implemented evenwhere the object(s) is (are) in motion, as preferred embodiments of theinvention utilize the projection of a single coded pattern. At each timeinterval determined by the imaging apparatus, only a single projectionof the encoded light pattern is needed to capture surface points ofobjects, thus enabling successive images to capture changing surfacepoints of objects in motion. That is to say, objects in the scene and/orthe camera and projector system may move in a three dimensional spacerelative to each other, thus enabling the dynamic depth capture of threedimensional objects in motion. As the object moves, the threedimensional coordinates of the object in a Euclidian space dynamicallychange. Thus, depth change over a series of captured images is reflectedin the movement or displacement of features along their respectiveepipolar lines for each captured image. As a result, different pointclouds are derived for each image frame.

In preferred embodiments, the imaging apparatus and the projector aretypically contained in a single device housing. The projected twodimensional light pattern may be projected by any light source,including but not limited to a video projector, a slide projector in thecase of a printed pattern, or laser/LED projectors. The use of singlepattern projectors in preferred embodiments allows for the straightforward use of diffractive projection, which leads to low powerconsumption, compactness, noise filtering due to narrowband type signalsand strong suitability to invisible NIR radiation applications. Theimaging apparatus may be a proprietary, or off the shelf, video or stillcamera typically with CCD or CMOS sensor. It is further understood thatthe epipolar lines in a captured image may be of nonlinear shape and thelinear lines in FIGS. 1A-1G and FIG. 3 are for illustration purposesonly.

Reference is made to FIG. 4, which is a simplified flow chart anddiagram illustrating how the corresponding 3D spatial locations ofidentified features in the captured 2D image are derived. The flow chartof FIG. 4 begins from the 2D image capture stage, stage 74, also seen inFIG. 2. Three different exemplary feature types, (A), (B), and (C),share an epipolar line in the captured 2D image 84 which includes boththe imaged object in 2D and the reflected pattern. The exemplary featuretypes in the coded pattern are projected onto an imaged object and allare subsequently observed along a single epipolar line E₁ in 2D image84, seen as step 76. The next step involves corresponding between eachidentified feature along the epipolar line and the originating featurein the projected 2D coded light pattern (step 78 a). This correspondenceis used to calculate, through triangulation methods, the 3D point inspace, or depth coordinate, from which the feature was reflected (step78 b). Notably, steps 78 a and 78 b are combined and seen as step 78 inthe flow chart of FIG. 2. The sum of all such triangulated featuresgives rise to a 3D point cloud (step 80) of the imaged object. Inpreferred embodiments, the point cloud 80 is further processed to obtaintexture and surface for the 3D image/video image, step 82.

Since the feature types shown are distinct, all three may be readilycorresponded to respective features in the projected pattern. If therewere more than a single appearance of say (B) on the epipolar line, thenambiguity would result when attempting to associate each appearance of(B) to the corresponding appearance in the projected pattern. Ambiguityas a result of multiple appearances of feature types on a singleepipolar line would perforce lead to triangulation errors and falsemeasurements of spatial locations on the imaged object. The embodimentsof the present invention therefore ensure at most a singular appearanceof any given feature type on an epipolar line, herein referred to asepipolar separation, to ensure a one to one matching process between anobserved feature and the feature's appearance in the pattern.

Again, in the present embodiments, the imaged object may be still or inmotion. If the imaged object is in motion, then the steps 74-82 arerepeated to provide 3D video. Alternatively, the imaging and projectorapparatus may be in motion relative to the object. In dynamicapplications, triangulation equations are computed at video frame ratesto give real time three dimensional coordinates of the object(s) inmotion.

Furthermore, as shown in the figures and described below, as the totalnumber of distinguishable epipolar lines in the captured imageincreases, the encoded light may be projected in a manner that allowsfor more appearances of features in the captured image. The total numberof distinguishable epipolar lines may increase due to many factors,including but not limited to: feature structure or shape, a low noiselevel as a result of a high resolution sensor, optical characteristicsof the system or more accurate coordinate detection algorithms.

FIG. 5 shows further exemplary point clouds derived through the methodsof embodiments of the present invention. Illustration 86 shows a frontalview of a point cloud of person waving. Each point in the cloud has athree dimensional (x, y, z) coordinate in space. Illustration 88 shows aside view that provides another depth perspective.

Reference is made to FIG. 6A. FIG. 6A now, showing a schematicrepresentation 500 of the appearance in an image of a reflectedspatially encoded pattern after being projected in accordance with theepipolar separation principle discussed above. That is, a light patternis constructed and projected in such a manner that any given featuretype appears at most once on any given epipolar line in the capturedimage.

In the present embodiment, the singular appearance of any given featuretype on an epipolar line is achieved through the orientation of theprojected periodic light code at a suitable tilt angle with respect tothe direction of the epipolar field. The epipolar field is vertical inthis example and denoted by epipolar lines 102A, 102B, and 102C. Theschematic diagram 500 of the reflected light pattern shows a periodicalor cyclic tiling of identical 10×2 matrices 90. Each matrix has twentyfeature types A-T. The matrix pattern repeats itself over the length andwidth of the image.

Now, when the pattern projection is rotated at an angle specified as 94in the figure, the periodic pattern of feature types in the imagerepeats itself on any given epipolar line over length H₁. Thus, for eachdistinguishable epipolar line over the imaging apparatus vertical fieldof view H₁, any feature type appears at most once. By tilting theprojection angle of the pattern, the separation of same feature typesonto distinguishable epipolar lines is achieved. For example, features96, 98, and 100, which repeat in the y-direction of the pattern everyother feature, are separated onto separate distinguishable epipolarlines 102A, 102B, and 102C, in the captured image. Resultantly asignificant number of features can be identified over the entire imagearea. The total number of identified features in a captured image isreferred to as image capacity or coverage.

Thus, epipolar separation of the tight periodic code allows for a largenumber of identifiable features to be captured in the image leading tohigh resolution. Furthermore, the periodic nature of the pattern allowsfor a coded pattern containing a small number of repeating featurestypes, thereby enabling each feature to remain small in size and furthercontributing to high resolution images. As an example, on a typicalsensor with 640×480 pixel resolution, each feature may have anapproximate area of 10 pixels. This translates into an image capacity ofapproximately 31 thousand features for every frame, and thus 31 thousand3D spatial coordinates.

In the above and current embodiments, for simplicity and clarity ofexplanation, width of pattern features in the 2D image is notconsidered. As such, each feature is associated with an epipolar line asif the feature has infinitely small size, and each feature is onlydistinguishable by one epipolar line. In actual practice, rather than asingle epipolar line being associated with an individual feature, thespatial formations that comprise the feature lie on distinguishable,substantially adjacent epipolar lines.

Returning to the FIG. 6A, distances denoted as D₁ and D₂ are equidistantdistances between distinguishable epipolar lines. The known horizontaldistance between distinguishable epipolar lines is a function of theimaging apparatus and image processing device and aids in the definitiveidentification and verification of reflected features in the image. Forexample, a reflected feature type (A) 98 is detected in the capturedimage at a certain y coordinate along epipolar line 102B. Thus, thenearest feature type (A), if indeed captured in the obtained image, canonly be found on the nearest distinguishable epipolar lines to the rightand left of epipolar line 102B, namely 102A and 102C. Although allepipolar lines are precisely vertical in the figure for illustrationpurposes, other embodiments are possible wherein the epipolar lines maybe lines or curves with varying x and/or y coordinates.

The limiting epipolar line separation factor is the minimum horizontaldistance necessary to distinguish between separate epipolar lines in thecaptured image. The ability to differentiate between features onseparate epipolar lines is based on several factors. Firstly, afeature's shape often determines whether the feature has a detectablecoordinate location after reflection from an imaged object. A codehaving features that can be assigned coordinates with greaterexactitude, allows for differentiation between features on ever closerepipolar lines. The object surface type also may have an effect, asfeatures from metallic surfaces, glass, and other transparent or verydark surfaces, for example, are reflected with less exactitude. Lowprojector SNR and inferior sensor optics may limit the distance betweendistinguishable epipolar lines.

Reference is now made to FIG. 6B. As has been explained until now, aslong as identical feature types fall on separate distinguishableepipolar lines, each feature type may be identified in the capturedimage without concern of confusion between them. However, under certaincircumstances now to be discussed, where reflected features are known tobe found only on pre-defined sections of an epipolar line, thenseparation of same feature types is even possible along the sameepipolar line. Two individual features of the same type on the sameepipolar line may appear on only limited sections thereupon, in the casewhere the imaged object moves within a very limited depth range over aseries of images. That is to say, if the imaged object movement isconfined to a particular distance range from the imaging system, thenmultiple occurrences of the same feature type may occur over a singleepipolar line. If the movement of an imaged object in 3D space isslight, then the corresponding movement of reflected features alongtheir respective epipolar lines is small. In such instances, it ispossible to separate multiple appearances of a given feature type on asingle epipolar line, where each feature type appears in a pre-definedrestricted range along the epipolar line.

To illustrate, we now turn to FIG. 6B, showing the same schematicrepresentation 500 as in FIG. 6A. Epipolar lines ρ,

, Ø, θ, φ, and β are shown and the image height is now larger, denotedH₁, to include multiple appearances of the same feature on a givenepipolar line in the image. On epipolar line

, for instance, features 96 and 506 are two individual features thatappear on the same epipolar line and are of identical type, namely (A).If each of these features is free to appear on any part of the epipolarline

in the image, then confusion may arise between the two features duringfeature identification. As explained, the correspondence between each offeatures 96 and 506 and the feature's unique appearance in the projectedpattern would be subject to error. But in the current embodiment, theappearance of feature 96 is guaranteed to appear only on subsection 96Aof epipolar line

, while the appearance of feature 506 is guaranteed to appear only onsubsection 506A of the same epipolar line

. In such a case, the distance between the two subsections, D

, is large enough so that no concern of confusion between same featuretypes exists. During the feature identification process, an (A) typefeature found on section 506A of epipolar line

in a given image is definitively identified as feature 506, while an (A)type feature found on section 96A is definitively identified as feature96. Regions 96A and 506A are two well-defined, spaced apart regions.Likewise, the appearance of features 98 and 504 upon epipolar line Ø isonly within ranges 98A and 504A respectively. Finally, features 100 and502 also only appear on respective ranges 100A and 502A along epipolarline θ. In all of the previous three cases, a large enough distanceexists between the ranges along a single epipolar line upon which twoidentical feature types may fall. Therefore, identification of twoseparate features of the same type is possible.

One way of having more than a single appearance of the same feature typeon an epipolar line is by increasing the image height to H_(1′) from H₁.The increase of the image height results in multiple appearances of thesame feature type along epipolar lines in the image. The multipleappearance of same feature types is due to the periodic cycle of thepattern, which repeats itself along the length of the enlarged H₁, orH_(1′). Alternatively, if the resolution of the image sensor isincreased and the pattern features are decreased in size, then featuretypes may repeat themselves along epipolar lines. As long as the depthmeasurement is limited to a certain range over which the imaged objectmoves, any given feature along the epipolar line only displaces a shortdistance for every frame. Thus, even two identical features may bedifferentiated on a single epipolar line, so long as the sections of theepipolar line upon which they shift are limited and spaced far enoughapart. The division of the epipolar line into predefined sections is afunction of the depth range limitation. Through such depth rangelimitations, more points are sampled in the image and resolution isfurther increased as the number of depth readings in the captured imageis greater.

We now refer to FIG. 7, which is a simplified illustration of a capturedimage of a preferred light pattern projected in accordance with epipolarseparation to ensure that same feature types fall on distinguishablyseparate epipolar lines. The figure illustrates the geometric principlesthat provide a significantly high number of identifiable featuresthrough projection of preferred patterns at angles to the epipolarfield. The angling of the projected periodic pattern ensures that samefeature types are captured on separate distinguishable epipolar lines.The features are preferably comprised of unique combinations ofspatially arranged sample points, and the number of sampled points inthe captured image is referred to as the image capacity or coveragefactor.

The question that could be asked is: Why not construct a code having asmany feature types as needed to cover the image frame, denoted by sensorheight H, without the need for epipolar separation? The answer lies inthe fact that single pattern imaging systems strive to have featuretypes as small as possible, thus providing higher sampling density perimage. To this end, in spatially encoded methods and systems, it ishighly desirable to encode the projected light with a minimal amount offeature types, since each feature type is represented by a spatial codeof some finite area. That is to say, the less features in the code, thesmaller pixel area needed to encode a feature, as each feature appearson the image sensor as light intensity formations over a certain numberof square pixels. By limiting the number of feature types through arepeating code, the number of pixels needed to represent a given featuretype is minimized. As such, the number of identifiable features in the2D image is increased, and thus the number of corresponding point cloudpoints, or point cloud density, is increased as well. A higher number ofpoint cloud points leads to higher resolution 3D images.

As such, it would not be effective or desirable to encode a large amountof feature types in order to obviate the epipolar line separationtechniques employed in the present embodiments. Through the techniquesof epipolar separation described in the embodiments, a rather smallmatrix code can provide “inflation” of the image capacity, as moreuniquely identifiable features may be observed for each captured imageframe.

FIG. 7 illustrates a schematic image of a projected coded pattern 104comprised of a series of 10×2 matrices of feature types, wherein theprojection is again at an angle to the epipolar field. Each feature ofthe pattern has square pixel size C×C. As such, the length of eachperiodic matrix in the pattern is XC=10C, while the height is YC=2C. Thedistance P represents the horizontal distance in pixels betweendistinguishable epipolar lines in the image. As seen, the projectedpattern 104 is rotated at an angle 106 with respect to the epipolarfield. It is shown that every other feature in the Y direction of thepattern, or in other words every reoccurrence of the same feature, fallson a distinguishable epipolar line. It is appreciated that the matrixmay take on various sizes, and this size is only an example forillustration purposes. Using the principles known in geometry of similartriangles, in can be proven that the triangle with sides H-U-V and thetriangle with sides YC-h-P are similar triangles, and thus the followinggeneral equation holds:H/U=YC/P where U=XC→H=XYC ² /PH is thus the number of pixels appearing on epipolar line 108. As theepipolar separation P is decreased, angle 106 becomes smaller. From theequation we see that, for a constant pattern area size, if only theepipolar line separation P decreases, then H grows larger.

For example, feature 110, an (A) type feature, occurs only once, asdesired, over the epipolar line 108. Feature 112 represents the nextoccurrence of feature type (A) after feature 110, and represents theupper limit of the image height. If the distance P betweendistinguishable epipolar lines is decreased, angle 106 is decreased inturn, and the same coded pattern with the same number of features isthus rotated less from the epipolar field. Identical features are nowseparated by closer epipolar lines that are still distinguishable, andthe effect of rotating the projected pattern to obtain epipolarseparation becomes more apparent. Feature 110 repeats itself over anever greater distance on epipolar line 108, thus expanding the value ofH and thus the number of uniquely identifiable features, or resolution,in the image. The lower limit of angle 106 occurs when epipolar linesare no longer distinguishable, that is to say, the same feature typefalls on two adjacent epipolar lines that are too close to accurately bedistinguished.

From the equation, we see that so long as the overall matrix size doesnot change any combination of X and Y dimensions of the matrix has noeffect on the image coverage. For instance, if the Y component of thematrix code is increased and the X component decreased, say from a 10×2rectangle, as shown, to a 5×4 square-like shape, we now have a matrixshape closer to a square rather than a long skinny rectangle. However,the overall matrix area (XY) remains the same. The angle 106 isdecreased and the pattern is rotated less from the epipolar field toensure epipolar separation on each distinguishable epipolar line. Asseen from the equation H remains the same, and thus the image coverageis not increased.

Notably, the matrix structure is such that the length of periodicity inthe X direction is much greater than that in the Y direction. Such amatrix code structure is referred to herein as preferred directioncoding. It is advisable for X to be much larger then Y in terms ofpractical optic assembly, calibration, operational physical andsensitivity. For example, vibrations or other shaking of the imagingsystem can cause a very small rotation angle 106 to become too small forepipolar differentiation. Furthermore, a smaller angle of rotation as aresult of non preferred direction encoded patterns requires decreasingthe image field of view by a larger safety factor in order to ensureepipolar separation.

Reference is now made to FIG. 8, which is an illustration of thecaptured image of FIG. 6A with reflected light pattern 500 now projectedat a smaller rotation angle 114 with respect to the direction ofepipolar field. The distance between distinguishable epipolar lines isdecreased to d_(I). Features 116, 118, and 120, are identical featuretypes that repeat themselves every other feature in the y-direction ofthe pattern. These features now fall on epipolar lines 122A, 122B, and122C. According to the equation described in FIG. 7, since the patternhas not changed, meaning X, Y, and C are constant but P decreases, Hbecomes larger. Therefore, H₂ is a larger vertical image height than H₁of FIG. 6A, providing more sample points and thus higher resolution inthe captured image. The imaging system can now sample more points in thecaptured image before encountering ambiguity, meaning without possiblyencountering the same feature type along the same epipolar line.Alternatively, instead of increasing the image field of view, a patternwith smaller features may be projected using a high resolution sensor.

When angle 114 is decreased too far, however, undesirable systemambiguity occurs. When angle 114 becomes too close to 0 degrees, anyslight movement or shaking in the calibrated projector/imaging apparatusmay cause a distortion and obstruct clear epipolar line distinction. Inaddition, as the periodic matrix in the pattern decreases in the Xdirection and increases in the Y direction, such as the almost-squareshape mentioned above, the projection angle needed to ensure that H isnot decreased, requires that P decrease to a point which is too small toprovide a safe horizontal distance for epipolar line distinction. As aresult of the above geometric factors and additional factors to accountfor possibility of error, the code preferably utilizes a preferreddirectional code in the X direction. Although the present embodimentutilizes a rotated pattern to achieve the result of epipolar separation,other embodiments including, but not limited to, skewing the projectedpattern exist.

Although in the previous embodiments the projected pattern is periodic,it is understood that in alternate embodiments other coding schemes arepossible, including, but not limited to non-ordered and/or non-periodiccoded patterns that may be projected to achieve epipolar separation. Theprevious embodiments have described spatial coding techniques using auniquely structured spatially encoded pattern P1. Nonetheless, it isappreciated that further embodiments may include temporal coding,spectral coding, a combination thereof, or any other two dimensionalcoding scheme or combination thereof, including combinations involvingspatial coding embodiments such as those described herein, that enabledifferentiation of a plurality of feature types along distinguishableepipolar lines.

Reference is made to FIG. 9, which is schematic diagram of a non orderedand non periodic pattern. Each feature type of a bi-dimensional spatialformation in a non-periodic code is represented in the pattern as anumber. It is appreciated that each feature type, numbers 1 to 6 in thefigure, appears not more than once on any give epipolar line. Exemplaryepipolar lines are 124, 126, and 128. An exemplary spatial coding shownis a pattern similar to dominoes. The spatial code may, of course, takeon many forms, possibly similar or identical to the code seen in FIG. 3and FIG. 10. Other structured light coding techniques may includebi-tonal, grey level, and/or multi-spectral techniques.

Reference is now made to FIG. 10, which are illustrations of exemplaryfeature types that comprise preferred encoded pattern P1. Features A-Jare examples of binary spatially coded feature types. Each feature iscomprised of a spatial combination or formation of five black and whitepoints, made up of a black (local minimum) or white (local maximum)center point 130 and four black or white peripheral or “saddle” points132. All possible combinations of the five black and white points leadto an alphabet of 32 unique characters or feature types. The exemplaryfeature types in FIG. 10 are arbitrarily named A-J, each correspondingto a different combination of a center point and four saddle points. Afeature type having a white center point corresponds to a local maximumillumination intensity in the projected pattern, while a feature typehaving a black center point corresponds to a local minimum illuminationintensity in the projected pattern. The peripheral points of each lettercorrespond to illumination intensities that are neither maximum norminimum, where a white peripheral point is closer to maximum intensitythan minimum intensity and a black peripheral point is closer to minimumintensity than maximum intensity. It is noted that other embodiments mayuse features having combinations of more or less saddle points. Forinstance, if each feature contained six saddle points, an alphabet nowexists of 128 unique characters or feature types and thus a largerpattern period. Such encoded features would be suitable for applicationsof larger patterns with more sampling points, such as high-resolutionmega-pixel imaging.

To illustrate further, reference is made to FIG. 11, which is anillustration of the exemplary pattern P1 as seen in an image I′_(P1)after being reflected from an imaged cube 134. The periodic patterncomprises the features types described in FIG. 10. A close-up of theimage showing one part of the cube is shown at the bottom of the figure.White center point 136 represents a maximum intensity reflection withinthe immediate surrounding pixel area of the image. Black center point138 represents a minimum intensity reflection within the immediatesurrounding pixel area of the image. White saddle point 140 shows areflection intensity that is closer to a maximum than a minimum. Blacksaddle point 142, is the opposite, and shows a reflection intensity thatis closer to a minimum than a maximum.

When features of the projected pattern are reflected from an imagedobject onto an image sensor, each of the five points of any givenfeature becomes a sample point in the captured image. These samplepoints are contained on adjacent sensor pixels upon which the reflectedfeature has been imaged. It is understood that whether a particularsample point is a maximum, minimum, or saddle is dependent on theprojected light intensity on that point as well as the reflectanceproperties of the material of the imaged object at that point.

The maxima, minima, and saddle points are preferably extracted throughidentifying their derivatives in the image. This means that they aredetermined through local variations in illumination. For example, awhite center point of a projected feature, representing a local maximumillumination intensity in the projected pattern, receives a low pixel(s)intensity or grey-value on the imaging sensor if reflected off of a darksurface. If such a center point were decoded independent of the centerpoint's surrounding sample points, the decoding engine may mistakenlyinterpret the low intensity value as indicating a local minimum, orblack center point. However, when analyzing the surrounding pixels of awhite sample point reflected from a dark surface, the decoding enginewill see pixel values of even lower intensity, and thus a correctidentification of the sample point as a local maximum occurs.

Likewise, a black center point of a projected feature, representing alocal minimum illumination intensity in the projected pattern, receivesa high pixel(s) intensity on the imaging sensor if reflected off of abright surface. Again, a direct intensity evaluation of such a samplepoint would falsely identify a feature containing a local maximum, orwhite center point. However, by measuring the changes in local intensityaround such a point, the decoding engine will recognize pixel values ofeven higher values. As a result, a correct identification of the samplepoint as a local minimum occurs. Therefore, when analyzing an image ofthe reflection of a single projected pattern critical point detection isused. Critical point detection means that changes in local illuminationintensity by known derivative analysis methods are used rather thandirect intensity evaluation to ensure correct feature detection in thedecoding process.

Furthermore, in the periodic repeating code of the previous embodiments,the relative horizontal distances between saddles and minima or maximaof features is constant. During the decoding process, the vector, or setof distances between elements or components of features is determinedand a constant vector is expected. Such a vector enables verification ofaccurate feature readings. False readings that arise from noise andtextural differences on the imaged object can be minimized through theverification process of distances between points of the feature. If, forinstance, a distance between a saddle and a minimum is larger than theexpected difference, then the decoding system may decide that anidentified feature element is merely noise rather than a part of afeature.

Known values of neighboring saddles and maxima are further utilized forthe feature validation process and error correction, possibly by thecross referencing of adjacent point grey values. For instance, in theidentification and validation process, certain characters within theknown alphabet may be eliminated if their spatial value arrangementcontradicts those of known neighboring characters. Another method fordecoding feature identity is by the integration of semi validatedcharacters to known groups of characters. An example case is where oneof four saddles is not clearly identified, but the other three have beenvalidated. It is understood that many different implementations ofcharacter recognition are possible, including wavelength analysis or anyother methods known in the art.

It is appreciated that the encoding scheme is not limited to a binarycode, nor to any preferred optical implementations of binary code, suchas bi-tonal, monochromatic, or bi-polar projection, and as in severalexamples given in previous embodiments above may be implemented in otherways, including but not limited to spectral coding, non periodic coding,and temporal coding.

Reference is now made to FIGS. 12A and 12B, which illustrate theconstruction process and the encoding scheme inherent in pattern P1.First referring to FIG. 12A, the process begins with a Debruijn sequence144. The sequence 144 is created over a character space S Σ={0,1,2,3}.The series has a length of two, meaning that every sequence of twonumbers appears at most once. Therefore, the length of the DeBruijnsequence is |S|=|Σ|²=4²=16.

Matrix 146 is generated from matrix 144 by replacing each number inmatrix 144 by the matrix's binary representation, where each binarynumber is written column wise. Matrix 148 is obtained from matrix 146 byflipping even columns; in other words, binary values in the upper row ofeven columns of the matrix are moved to the bottom row and visa versa.The flipping of even columns in matrix 146 is performed to avoid asituation of code duplicity, meaning to ensure that no letter appearsmore than once in any given cycle of the matrix in the pattern.

We now turn to FIG. 12B. The matrix 148 of FIG. 12A is mapped into a twodimensional pattern of rhombi 150, having two colors of grey, each colorrepresenting a 1 or 0 in matrix 148. This two color grey pattern isrepeatedly mapped onto a pattern 152 of black, white, and grey rhombi.The resulting pattern 154 is formed, having four colors of grey. Everygrey rhombus of pattern 150 colors one of the grey rhombi in pattern 152one of two shades of grey. The black and white rhombi remain unchanged.In the next two steps, the binarization of rhombus pattern 154 iscarried out, possibly by Gaussian smoothing followed by obtaining whatis referred to as a sharp intensity threshold, although other techniquesare possible. The resulting pattern P1, as shown initially in FIG. 3,provides a bi-tonal sharpened pattern for projection that allows theepipolar separation techniques described above.

It is understood that the encoding described herein is a mere single wayof constructing a viable two dimensional spatial formation-based lightcode having multiple feature types to be used in preferred embodimentsof the present invention and other code types, such as Perfect Maps, MArrays and pseudo random codes are possible.

Reference is made to FIG. 12C, which are simplified illustrations of thesample points in the pattern P1 after being reflected from an imagedobject and viewed as part of a captured image in an imaging apparatus.Pattern 164 is a close up of pattern 154 of the previous figure. Thethirty two binary values of matrix 150 are seen as thirty two saddlepoints 168. Each of these thirty two saddles points becomes a samplepoint. Each group of 4 saddle points has a black dot 170 or white dot172 in the center. The black and white dots are black and white rhombiin pattern 154, and become sample points as well. Altogether, there are64 sample points in every cycle of the code in the pattern. These pointsare shown in the brightened area 173. Pattern 166 is a close up ofpattern P1. In pattern 166, the sample points are seen after the patternis processed with Gaussian smoothing and sharp intensity threshold.

Reference is made to FIG. 13A, which illustrates a preferred projectionand imaging method that enables epipolar separation techniques describedabove. Once again, the encoded light pattern P1 of previous embodimentsis shown. The encoded light pattern is seen here, as in FIG. 3, rotateda certain angle in relation to the epipolar line 174 in the capturedimage, thus placing same feature types of the code on separate epipolarlines. The angle of rotation is described by the following equation.Sin⁻¹(P/YC)

By selecting the angle of rotation, the epipolar separation techniquesdescribed above may be carried out. Calibration methods known in the artare used to determine the exact location of the epipolar lines in thecaptured image. Furthermore, to ensure that no two identical features ofthe code repeat themselves on a given epipolar line, the projectedpattern height H_(P) is limited in the projector apparatus, asexemplified by the brightened area 175 in the pattern. As a result, eachdiagonal column for every period is cut by a different epipolar line.Each epipolar line in the captured image therefore contains one and onlyone feature of each type. If the measured depth range is limited, thenas discussed above in FIG. 6B, more than one of the same feature typecan appear on each epipolar line in the image. The image height H inpixels can be described by the equation in FIG. 7 above. Since theaspect ratio of the shaded area is 4/3, the total surface area in termsof sampled pixels is defined by the following equation:A=H ²(4/3)=X ² Y ² C ⁴ /P ²(4/3)

Reference is made to FIG. 13B1. For certain 3D imaging applications, alarger number of sample points is needed to obtain the desiredresolution. For such a purpose, a variant coding scheme of the patternP1 in above embodiments, named P2 herein, may be utilized. The P2pattern is shown in FIG. 13B1. As in P1, P2 is a binary spatially codedperiodic pattern containing feature types comprised of spatialcombinations of five black and white elements or points. Likewise, as inP1, the black (local minimum) or white (local maximum) center points aresurrounded by 4 saddle points. In pattern P2, however, two geometricalvariants of each letter are contained over each cycle of the code andthus the population of letters is doubled from 32 to 64. That is to say,each letter has one variation Y1 in which the center point is shiftedtowards the left and falls closer to the left saddle points and anothervariation X1 in which the center point is shifted towards the right andfalls closer to the right saddle point. The alternating maxima andminima do not fall one on top of the other in the vertical direction asin the coded pattern P1, but rather each maximum is shifted slightly tothe right or left of the minima directly above and below. Similarly,each minimum is shifted slightly to the right or left of the maximadirectly above and below. Such a pattern creates a “zig-zag” typeeffect, where alternating rows are shifted either slightly to the rightor to the left. In all, in a given cycle, there are 32 black minimum, 32white minimum, and 64 saddles that in total comprise 128 sample points.

Notably, the relative horizontal distances between points of eachfeature in the image, as discussed above, now allow for a vector thatrepresents an encoding. That is to say, in addition to the five pointsof the feature which allow for a 2⁵=32 feature code, the varyinghorizontal distances between points of a given feature allow foradditional feature types to be present in each pattern, thus adding tothe number of sample points possible for a given image. Variouscombinations of horizontal distances of the same five feature points arepossible.

We now turn to FIG. 13B2. Pattern P2 is constructed in a similar way asdescribed above in FIG. 12A, 12B using a Debruijn sequence. A binarymatrix based on a slightly modified Debruijn sequence is mapped into atwo dimensional pattern of rhombi 282 having two colors of grey. Thistwo color grey rhombi pattern is repeatedly mapped onto a pattern 284 ofblack, white, and grey rhombi. The resulting pattern 286 is formed,having four colors of grey. Every grey rhombus of pattern 282 colors oneof the grey rhombi in pattern 284 one of two shades of grey. The blackand white rhombi remain unchanged. In the next two steps, thebinarization of rhombus pattern 286 is carried out, possibly by Gaussiansmoothing followed by obtaining what is referred to as a sharp intensitythreshold, although other techniques are possible. The resulting patternP2, as shown initially in FIG. 13B1, provides a bi-tonal sharpenedpattern for projection that allows the epipolar separation techniquesdescribed above.

Graphical processing known in the art is carried out on the rhombusarray to obtain the “zig-zag” pattern that allows for two shiftedvariants of each letter as described above.

Again, the rotation of the grid pattern ensures that every two identicalfeatures that appear in an alternating fashion on each diagonal columndo not fall on the same epipolar line.

The process of decoding is characterized by comparing relativehorizontal distances between the various sample points, meaning either aminimum, maximum or saddle point. As described above, the decodingprocess attempts to identify locations of maxima, minima and saddlepoints. The horizontal distances between such points serve to validatethe existence of a feature at that location in the 2D image. In thepresent embodiment, these horizontal distances may be utilized foridentification of features in such a zig-zag pattern. For example, bydetermining relative horizontal distances between an identified maximumand the maximum's associated saddle points, the identity of the lettercomprising such an array of points may be determined.

Reference is made to FIGS. 13C and 13D. In FIG. 13C, two patterns T1 andT2 are presented. T1 is simply the pattern P1 in previous embodimentsdoubled in height. Pattern T1, like P1, contains 32 features and has aperiodic cycle of 16. A captured image containing projected pattern T1is shown as I_(T1) and allows for two cycles of the pattern in thevertical direction. Thus, each feature appears twice on a given epipolarline in image I_(T1). Noticeably, these features are at a distance fromeach other that is equal to the height of pattern P1.

T2, being equal in height to pattern T1, is a periodic spatially encodedpattern having unique bi-dimensional formations different than T1. Thepattern T2 contains 32 types of features. Each feature type appearstwice consecutively in the cycle (other than 4 unique feature types thatappear once in each cycle). Each pair of adjacent identical featuretypes shares common saddles. For example, in FIG. 13D, features 350 aand 350 b are of the same type and appear adjacent to each other. Eachfeature is comprised of an extremum and four saddles. Two of the foursaddles for each feature are common to both features. Therefore, eachpair of adjacent features contains two extremas and six saddle points.Each sequence of two identical features repeats itself every 32 lines inthe pattern.

As a result of such a repetition within the pattern T2 cycle, when thepattern is projected at a rotation to the angle of the imaging sensor,feature types reappear within close proximity on the same epipolar linein the resultant obtained image. Referring to FIG. 13D, such an exampleis seen on epipolar line 354 where identical features 350 a and 350 cappear on the line in close proximity. As seen in the figure, identicalfeatures 350 a and 350 b are adjacent to one another in the cycle.Therefore, the rotation places like feature types in close proximity onthe epipolar line. Likewise, identical features 360 a, 360 b, and 360 cappear in a group of three as seen in the cycle. These three featuresthus appear along epipolar line 362 in close proximity as shown. Thelocation range of each feature type along the epipolar line is thusknown.

Returning to FIG. 13C, we see that an advantage of enlarged pattern T1is the enlarged number of sample points that may be imaged andcorrelated to depth points in space. However, the appearance of likefeature types on the same epipolar line presumably causes ambiguity inthe decoding process as discussed above. For instance, identicalfeatures 366 a and 366 b are on the same epipolar line 370 in I_(T1). Toresolve such ambiguity, a second pattern T2 may be projectedintermittently with T1. Since T2 has a cycle twice the length of T1,then in an image I_(T2) having the same height as image I_(T1), anidentical feature group 364 or 368 does not repeat itself over thelength of the image. Each feature group only appears once along thelength of the image I_(T2).

Now, when patterns T1 and T2 are aligned in the projector apparatus andprojected temporally, a comparison can be made between observed featureswithin a range along parallel epipolar lines in successive images. Thematching of features from a first image of I_(T1) to sister features ina second image of I_(T2) can allow for feature verification in the firstimage. As an example, feature 366 a is seen in a particular range alonga given epipolar line 370 in a first image of reflected pattern I_(T1).Likewise, feature 366 b appears along that same epipolar line in thefirst image I_(T1), albeit at a distance from 366 a. Such a situationmay lead to ambiguity in I_(T1).

A second image I_(T2) having reflected pattern T2 captured in successionto the first image contains identical feature set 364 a, 364 b, and 364c on the same epipolar line, 370, as feature 366 a appeared in imageI_(T1). Also, in a different region of that same epipolar line in imageI_(T2) appears identical feature set 368 a, 368 b, and 368 c. Sincethese feature sets are distinct on the epipolar line, each set can berespectively compared to features 366 a and 366 b. That is to say, bycorrelating feature 366 a from I_(T2) with features set 364 a, 364 b,364 c in I_(T2) and feature 366 b from I_(T1) with features set 368 a,368 b, 368 c in I_(T2), ambiguity can be annulled. This is because eachof features 366 a and 366 b in the first image is matched to a uniquefeature set in the second image.

The process of projection and decoding is as follows. T1 and T2 areprojected alternately, synchronized with the camera. The images aredecoded separately at first. The decoding of I_(T1) is carried out asdescribed in above embodiments, although each feature carries two imagecoordinates as it appears twice on the epipolar line. I_(T2) likewise isdecoded in the same manner as above embodiments and like features arefound vertical to each other along the same epipolar line. Therefore, ifno movement in the imaged objects occurred between frames, each featurewithin the image sequence I_(T1), I_(T2), I_(T1) can be verified bycorrelating between the two images. If I_(T1) and I_(T1) are identical,then the imaged scene is static for the imaging time frame. I_(T1) actsas a resolving pattern, as each feature at a specific location in imageI_(T2) is different than the feature at the corresponding location inimage I_(T1). Since multiple feature types of the same type reappear onI_(T2) in close proximity, even if movement occurs between frames, thematching process can still identify associated features between the twoimages. In essence, the above process combines principles from bothtemporal and spatial encoding methods.

In temporal coding techniques, wherein more than one pattern isprojected on the object for each captured image, it is understood thatlimitations may have to be placed on the speed of motion of the objectbeing imaged.

Reference is made to FIG. 14, which shows image I_(P1) and exemplarycharacters or feature types of pattern P1. Characters 176 representfeatures encoded over every period of the pattern, the nature of suchfeatures being described above in FIG. 10. It is noted that theassignment of letters to feature types in the current figure isindependent of FIG. 10. The length of each period of the pattern P1 is16 and the height is 2. That is to say, same feature types belong toeither the same diagonal column, for instance identical features A andE, or to columns modulo 16, for instance identical features A and F.Modulo 16 means that same features are distanced periodically every 16diagonal columns. Features from diagonal columns that are not modulo 16,however, are always of different types, such as features A, B, C, and D.On every diagonal column only two different feature types appear in analternating fashion; one feature having a black center value and onewith a white center value, for example A and G. As explained, featuresA, E and F are individual features appearing in the pattern that are ofthe same type. For purposes of clarity, these individual features areonly referred to as feature type (A) in FIG. 3 and other figures whereP1 is shown, while in the current figure, each separate feature isassigned a unique letter, A, E and F respectively.

Again, the rotation of the grid pattern ensures that every two identicalfeatures that appear in an alternating fashion on each diagonal columndo not fall on the same epipolar line. As an example, features A and Eare two individual features in the pattern that are of the same type.Feature A falls on epipolar line 178. Feature E, identical to feature A,is found four features below A on the same diagonal line. On thisdiagonal line, the A/E feature type occurs every other feature. FeatureE falls on epipolar line 180, horizontally differentiable from 178. Wesee that each feature in the image can be expressed in terms of featuretype and the feature's associated epipolar line.

The above embodiments discuss the projection of a single pattern onto animaged object. The projection of a second pattern can provide furthervaluable information in determining the third dimension or depth. Inparticular, dual projection together with dual imaging may significantlyincrease the accuracy and reliability of feature type identification aswell as provide new ways of reaching extremely high resolution in depthreadings. Moreover, in certain embodiments, texture information may beextracted from the information obtained from the dual projection methodsdiscussed herein.

Dual projection methods may include both temporal methods and/orspectral methods. Temporal methods are such where two or more patternsare projected over separate time intervals. Spectral methods are suchwhere two or more patterns are projected with differing wavelengths oflight. Temporal methods may utilize a spatial light modulator such as anLCD, DLP, or DMD. Spectral methods may, for example, be carried out byan optical multiplexer.

In the embodiments below, in the case of spectral-separationimplementations, spectral separation is preferably small so as to allowfor direct comparison of reflection intensities. That is to say, twoidentical patterns projected onto an imaged object at different butadjacent wavelengths will reflect with substantially identicalreflection intensities. So for instance, in a first image I′_(P1) a highintensity or white spot is projected onto a particular bright surface ata certain wavelength and is mostly reflected. In a second inverse imageP′_(−P1) a low intensity or black spot is projected onto the same brightsurface at an adjacent wavelength and is mostly absorbed. Since theblack spot in I′_(P1) is projected at an adjacent wavelength, the blackspot behaves in an opposite manner to the projected white spot inI′_(P1) for the given surface and is mostly absorbed. Likewise, whitespots projected at the two adjacent wavelengths in the images bothreflect in almost identical manners. Black spots projected at the twoadjacent wavelengths in the images both absorb in almost identicalmanners.

Reference is made to FIG. 15, which is an illustration of a preferreddual projection and dual imaging embodiment of the present invention.I′_(P1) is an image of a Rubik's Cube having pattern P1 of the previousembodiments, as explained in detail in FIGS. 3 and 11, projected uponthe cube. Image I′_(−P1) is a second image of the cube having thenegative of the pattern P₁ projected thereupon. Finally, the imageI′_(P1)−I′_(−P1) is the resultant image obtained from the subtraction ofI′_(−P1) from I′_(P1). Close-ups 182 and 184 are the respective patternreflections from the bottom right four squares of the cube in imageI′_(P1) and I_(P1). In both images I′_(P1) and I′_(−P1), an imaged whitepoint is a result of a projection and thereafter reflection of a maximumlight intensity on that surface location of the imaged cube. Likewise,an imaged black point is a result of a projection and thereafterreflection of a minimum light intensity on that surface location of theimaged cube. So, if in I′_(P1), a reflected white point is seen at aparticular surface location on the cube, then in I′_(−P1) a reflectedblack point is observed at the same surface location on the cube. Forinstance, white maximum point 186 seen in image I′_(P1) is seen as blackpoint 188 in image I′_(−P1). Likewise, white saddle point 190 in imageI′_(P1) is replaced by black saddle point 192 in image I′_(−P1).

As the Rubik's Cube contains squares of varying colors, the reflectedpattern shows high contrast when reflected from squares of brightcolors, such as 194, and less contrast when reflected from squares ofdarker colors, such as 196. This is a result of the fact that the energyof projected white spots, or light intensity maxima, is reflected frombright surfaces to a much larger extent than the energy's reflectionfrom dark surfaces, where much of the energy is absorbed. A projectedwhite spot, reflected from a bright colored surface appears white in theimage, signifying an area of the imaged object with high intensityreflection. In contrast, when a projected white spot is reflected from adark colored surface, the reflected white spot appears much closer toblack in the image, signifying an area of the imaged object with lowintensity reflection. In either case, however, a projected white spotwill always reflect from any given imaged object surface point at ahigher intensity than a projected black spot on that same point.

Therefore, no matter what the imaged surface reflectance characteristicsare, an imaged black point will always have a reflection intensity lowerthan an imaged white point at the same location on the imaged objectsurface. Thus, the subtraction of a black point intensity value 188 at aparticular location in image I′_(−P1) from a white point intensity value186 at the same location in image I′_(P1) will always yield a positiveintensity value 200. Likewise, the subtraction of a white point 202 inimage I′_(−P1) from a black point 204 in image I′_(P1), both at the sameimage location, will always yield a negative intensity value 206. Assuch, a value of each pixel in the resultant image is signed, eitherpositive or negative. For visualization purposes, as display screens andprints cannot understand negative pixel values, a normalized scale maybe used to represent both positive and negative resultant intensityvalues.

By projecting a pattern and the pattern negative on an object andsubtracting the image of the reflected negative pattern from the imageof the reflected original pattern, maxima and minima locations in imageI′_(P1) may be determined directly by a measure of the grey-level orintensity of the same image locations in resultant imageI′_(P1)−I′_(−P1). All positive intensity values in the I′_(P1)−I′_(−P1)image indicate a white spot while all negative intensity values in theI′_(P1)−I′_(−P1) image indicate a black spot. Again, since negativeintensity values may not be understood by display screens, a normalizedscale may be used for image display. This direct measure of maxima andminima with such a dual projection is unlike the single patternprojection case described above in FIG. 11. In the case of singlepattern projection, the additional analysis of local intensity values ofsurrounding points to the maxima and/or minima is preferred to ensure ahigher level of correct feature type identifications. Therefore, thesubtraction method of the present embodiment allows for a more robustpattern recognition analysis and thus a more reliable and powerfulengine for derivation of 3D spatial coordinates from identified featuresin the captured 2D image.

Furthermore, the dynamic range of the intensity values in theI′_(P1)−I′_(−P1) image is doubled by the subtraction of the twopatterns. This gives rise to a more accentuated pattern and thus aids infeature type recognition. Another advantage is the cancellation of theeffect of ambient light on the resultant image I′_(P1)−I′_(−P1). Theresultant image shows an accentuated pattern with less interference ofimaged object texture.

It is understood that the two images may be taken temporally or at thesame time through use of spectral differentiation. Likewise, the twopatterns may be projected with spectral or polarity separation, at thesame time or temporally, or both patterns may have the same wavelengthor polarity but the projections are separated in time. So long as thespectral separation is close, the reflection intensities of the tworeflected patterns will be virtually exact negatives at any given imagedobject surface location.

Reference is made to FIG. 16, which is an illustration of the additionof the two images, I′_(P1) and I′_(−P1). The addition of the two imageswith opposite pattern illumination intensities projected thereupon leadsto a cancellation of the pattern in the I′_(P1)+I′_(−P1) image. This isbecause the addition of any two reflection intensities for any givenidentical location in the two images leads to a resultant image showingthe reflection of maximum light intensity from all points on the object.The variations in reflectance intensity are thus only a function of thetexture of the imaged object. That is to say, the changing texture ofthe imaged object affects the reflection intensities with which a whitepoint is reflected from the various points of the imaged object. Theresultant image separates the texture of the depth imaged object fromthe imaged pattern observed in images I′_(P1) and I′_(−P1).

Reference is now made to FIG. 17, which is a preferred embodiment of adual projection and dual imaging method of the present invention.I′_(P1), as in previous figures, is an image of a Rubik's Cube havingthe pattern P1 projected upon the cube. Image I′_(−C) is a second imageof the cube having the same pattern as in I′_(P1) projected thereupononly the maxima and minima are switched. That is to say, a maximum 186in image I′_(P1) is seen as minimum 210 in image I′_(−C). However, asaddle point, for instance white saddle 190 in image I′_(P1) remainswhite saddle point 212 in image I′_(−C). Thus, a maximum is replaced bya minimum and visa versa, but the saddle points remain unchanged in bothimages. Finally, image I′_(P1)−I′_(−C) is the resultant image from asubtraction of image I′_(−C) from image I′_(P1).

As seen in close up 214, what remains in the image I′_(P1)−I′_(−C) arethe maxima and minima, sometimes referred to as the carriers of thecode. Again, for any given location on the image sensor, a reflectedlocal intensity maximum point in image I′_(P1) is replaced by areflected local intensity minimum point in image I′_(−C). The localintensity maximum appears as a white spot in the image, while the localintensity minimum appears as a black spot. The saddles are identical inboth images. So, for instance, when the intensity value of location 210in image I′_(−C1) is subtracted from the intensity value of the samelocation 186 in image I′_(P1), white spot intensity value 216 isobtained in the resultant image I′_(P1)−I′_(−C). Similarly, when theintensity value of location 218 in image I′_(−C) is subtracted from theintensity value of the same location 220 in image I′_(P1) black spotintensity value 222 is obtained in the resultant image I′_(P1)−I′_(−C).In contrast, when saddle point 212 in image I′_(−C) is subtracted fromthe saddle point 190 of the same intensity and same image location, theidentity of the saddle points as white or black disappears. On the otherhand, the locations of the saddle points in the I′_(P1) image are nowsharpened and accentuated by the subtraction. The subtraction of the twoimages thus neutralizes the saddle values in the image but clearlydelineates their locations. Moreover, the subtraction of the two imagesleads to a cancellation of ambient light.

The resulting image I′_(P1)−I′_(−C) can now easily be scanned formaxima, minima, and saddle point locations. As explained above, in FIG.10, the maxima and minima in the imaged pattern represent the centerpoints of reflected features in the pattern. As the resultant imageshows only the saddles' location, but not value, the search for maximaand minima is thus simplified, and becomes faster and more error-proof.These maxima and minima then indicate the existence of a feature and thefeature's epipolar line location in the image may be determined. Whatremains is to determine feature type by identification of the saddlepoints. This may be done by either analysis of I′_(P1) as in the singlepattern case, or as described next.

Reference is made to FIG. 18, which shows the addition of the two imagedpatterns of FIG. 17, I′_(P1) and I′_(−C). The addition of the two imagesleads to a resultant image that shows the reflection of maximum lightintensity for all carrier locations. In other words, all former carrierpositions become white in the resultant image. Only the saddle pointsremain. The addition of the two images leads to the addition of twoopposite carrier reflection intensities at identical locations in thetwo images. In contrast, the saddles are identical in both images, sothe addition of the two images leads to an accentuation of the saddlepoints.

For instance, in close-up 224 of the I′_(P1) image, a black feature withcenter point 226 is shown. The feature's two upper saddles 228 and 230are black. The bottom left saddle 232 is also black while the bottomright saddle 234 is white. In close up 236 of the same place on theRubik's Cube in the I′_(−C) image, the same feature now has white centerpoint 238 instead of black. The saddles 240-246 remain the same.Therefore, the resultant image close-up 248 shows the four saddles250-256. Since all carriers become white in the resultant imageI′_(P1)+I′_(−C), the saddles may be clearly identified without confusionwith local maxima and minima. FIGS. 17 and 18 show that the separationof the carriers from the saddle points allows for a more robust decodingprocess as the detection of carriers and saddles is simplified. ImageI′_(P1)−I′_(−C) provides the carrier and saddle detection and locationinformation while image I′_(P1)+I′_(−C) provides the saddle identityinformation.

Reference is made to FIG. 19, which is an illustration of the resultantimage obtained from the absolute value of the subtraction of imageI′_(−P1) from image I′_(P1). The resultant image 270 is an outline ofthe pattern. Since the two images are inverses of each other, asdiscussed above in FIGS. 15 and 16, the individual patterns can beexpressed as two opposite sinusoidal curves. Graph 260 is a crosssection of the sinusoidal curves that comprise each of the two inversepatterns. Each local maximum and white saddle location in the I′_(P1)image is replaced, respectively, by a local minimum location and blacksaddle in the I′_(−P1) image. Conversely, each local minimum and blacksaddle location in the I′_(P1) image is replaced, respectively, by alocal maximum location and white saddle in the I′_(−P1) image. Theabsolute subtraction of any two extrema or saddles from identicallocations in the two images leads to a bright intensity spot in theresultant image |I′_(P1)−I′_(−P1)|. Therefore, any maximum, minimum orsaddle point is bright in the image |I′_(P1)−I′_(−P1)|. This is seen inclose up 272 of the resultant image.

The dashed curved lines 262 and 264 indicate the border areas betweenblack and white spots of the respective images I′_(P1) and I′_(−P1). Onthe sinusoidal graph, these dashed curved lines are represented by themeeting point, or the 0 point, between the respective graphs of imagesI′_(P1) and I′_(−P1). Where these points meet, the result of theabsolute subtraction |I′_(P1)−I′_(−P1)| is 0. These meeting points occurin a two dimensional plane in the resultant image, creating the blackoutline 266 of the pattern. The black outline represents a minimum inthe image 270 and provides a sub-pixel resolution of a continuity ofhighly identifiable curves in the resultant image. Points along thispattern outline can be associated with spatial coordinates throughvarious triangulation techniques.

Reference is made to FIG. 20, which is an illustration of the resultantimage obtained from the absolute value of the subtraction of imageI′_(−C) from image I′_(P1) As discussed in FIGS. 17 and 18 above, theI′_(−C) image of the cube has the same pattern as in I′_(P1) projectedthereupon only the maxima and minima are switched. That is to say, amaximum 186 in image I′_(P1) is seen as minimum 210 in image I′_(−C).White saddle point 190 in image I′_(P1), however, remains white saddlepoint 212 in image I′_(−C). Thus, a maximum is replaced by a minimum andvisa versa, but the saddle points remain unchanged in both images.

The absolute value of the subtraction of image re from image I′_(P1)yields image 274. The absolute subtraction of either a black or whitecarrier in image I′_(−C) from either a black or white carrier in imageI′_(P1) leads to all white carriers, seen in close up image 276. Theblack crossings represent the borderline areas between black and whitecarriers in the I′_(P1) and I′_(−C) images. The borderline areas are theareas where the white carrier begins to turn black and visa versa. Atthe precise borderline, the pixels are neither black nor white, as thevalue is precisely between a maximum and a minimum. The absolutesubtraction of these pixels leads to 0 value, or a black or minimumpoint in image 274. The resulting image has less ambient light andallows for clearer decoding and accurate localization of extrema.

Reference is made to FIG. 21, which is an illustration of a particularembodiment of dual pattern projection and dual pattern imaging. Thefirst image is I′_(P1), and the second is an image of the cube withambient light illuminating the image. The subtraction of the image ofthe cube with ambient light from the image of the cube I′_(P1) havingthe pattern P1 projected thereupon, provides a resultant imagesubstantially free of ambient light. As a result, the pattern in theresultant image is accentuated and thus more readily decoded. This isseen in close up of resultant image 278 where the black and white pointsof the image are clearer than seen in the original I′_(P1) image.

Reference is made to FIG. 22, which is an illustration of a particularembodiment of dual pattern projection and dual pattern imaging. Thefirst image is I′_(P1), and the second is an image of the cube withuniform or relatively uniform light illuminating the image. The divisionof the image of the cube I′_(P1) by the image of the cube with uniformlight projected thereupon, provides a resultant image substantially freeof object texture. As a result, the pattern in the resultant image isaccentuated and thus more readily decoded. This is seen in close up ofresultant image 280 where the black and white points of the image areclearer than seen in the original I′P1 image.

In all of the dual projection and imaging embodiments of FIGS. 15-22,the alignment, or relative spatial position, between the imaged objectin both images is limited somewhat. The degree to which movement betweenimaged frames is limited is typically a function of the imaging strategyand the speed of the imaging device. Both temporal and spectral imagingdevices may be used with the previous embodiments to capture dualimages.

For instance, the simplest implementation of temporal imaging of thedual patterns in the previous embodiments is through two “snapshots” ofthe imaging device over two time intervals. One advanced method oftemporal dual imaging, supported by modern camera boards, isnon-uniform-exposure timing. Modern camera sensors enable control overthe exposure time of the camera sensor. So, for example, it is possibleto provide pairs of interval exposure times placed close to each other,rather than uniformly spaced in time. For instance, image I1 may betaken at t=0 milliseconds, image I2 at t=5 milliseconds, image I3 at 100milliseconds, image I4 at 105 milliseconds, and so on. As a result, aseries of images is obtained at an average rate of 20 images per second.However, pairs of images are provided that are taken at consecutive timeintervals rather than spaced apart in time. As a result, if the movementoccurring between the first and second time interval is slight enough,or alternatively if the imaging device is fast enough, comparisonbetween the dual images may be carried out as in previous embodiments.

The above strategy of two consecutive images at adjacent time intervalsis dependent on fast projection speeds. The imaging apparatus sensorshould, ideally, not be exposed at any time beyond the projection time,as this would lead to ambient light disruptions in the imaged pattern.Other sensor designs to increase imaging speed for the embodiments ofthe present invention include certain CMOS per-pixel-based computationsensors. These specially designed sensors carry out computationsimmediately at each pixel location upon the pixel receiving lightintensity. This is in contrast to computing pixel values subsequent tothe emptying of the sensor to computation modules.

Furthermore, the above dual imaging strategy may be implemented toobtain texture images in parallel to the captured depth information. Inpreferred embodiments, the pattern is projected every other image frame.In the remaining image frames, uniform illumination is projected ontothe object and the object is imaged. Assuming minimal relative motionbetween the two frames, it is possible to apply the texture informationobtained while projecting uniform light onto the object to depthinformation captured in the images containing the projected pattern.

It is understood that spectral methods of capturing two patterns arealso possible. Such implementations may include the use of multipleCCDs.

Reflectance characteristics of certain high texture surfaces may lead tofeature identification error in single imaging device systems.Therefore, in order to ensure that correct 3D spatial coordinates arederived, an additional imaging device may be added to the system. Theadditional imaging apparatus allows for what is referred to as stereodepth decoding, in addition to the single image decoding methodsdescribed previously herein.

For example, a dual imaging system using methods and apparatuses ofcertain embodiments may be conveniently mounted on a moving vehicle inan urban environment to capture the geometric constructs in threedimensions of buildings, landmarks and other objects both stationary andmoving in an urban scene.

To further understand stereo depth decoding using the methods andapparatuses of preferred embodiments, we refer to FIG. 23. Instereo-based depth imaging and decoding systems, two imaging devicesplaced near to each other capture a given scene, preferablysimultaneously. The two images, denoted as image A and image B, arenearly the same. However, since the two images are taken from slightlydifferent angles, imaged objects appear at slightly different pixellocations in the two images. Given a fixed distance between the imagingdevices for every frame, any object that ever appears on a certain pixellocation in image A must appear on a specific epipolar line in image B.Likewise, any object that ever appears on a certain pixel location inimage B appears on a specific epipolar line in image A. The distance ofthe imaged object appearing at a certain pixel location in one imageaffects where along the corresponding epipolar line the object appearsin the second image.

Now, when a projector is added in between the two imaging devices, orcameras, a structured light pattern such as discussed above may beprojected onto the imaged urban objects. In each of the two images,reflected pattern features appear together with the texture of imagedobjects, such as seen in FIG. 11. A given pixel in either image maycontain part of a reflected feature and thus any randomly chosen squarepixel area of the image may contain features or parts of features inaddition to the imaged object texture. The features, or parts offeatures, together with the imaged object texture comprise the totalityof information on any given pixel. The information appearing at a squarepixel location in image A, will always appear on a unique set ofcoordinates (a specific set of epipolar lines) in image B. Likewise anyinformation appearing at a square pixel location in image B, will alwaysappear on a unique set of epipolar lines in image A. Further, ifepipolar separation techniques associated with the pattern structure andprojection are utilized, feature types appear only once on each epipolarline. Therefore, each small area along epipolar lines in each image isunique and the features provide non-repeating “artificial texture” foreach image. The set of points comprising epipolar fields between imagesis determined generally through stereo calibration or stereo rigcalibration techniques known in the art.

The epipolar line in image B, EP1B, comprises the totality of pointsupon which the information contained on pixel area PX1 in image A may befound in image B. Likewise, the epipolar line in image A, EP1A,comprises the totality of points upon which the information contained onpixel area PX2 in image B may be found in image A. The relationshipbetween pixel locations in image A to pixel locations in image B isreferred to as an epipolar field.

Referring to FIG. 24, two additional epipolar fields exist. These fieldsare: 1) the epipolar field between the projector and image A and 2) theepipolar field between the projector and image B. These fields behave asdescribed in previous embodiments above and are not expanded on furtherhere. Noticeably, in light of the above discussion, feature F1 mayappear in Image A at pixel area PX1 in addition to the texture of animaged object. In such a case, both feature F1 and the imaged objectappear at the same point on EP1″ in image B.

Stereo correspondence can then be carried out as follows. A particulargroup or “window” of pixels is chosen at a certain location in eitherimage, say image A. A similar group of pixels to that chosen in image Awill only be found along a certain set of epipolar lines in image B.Image B is scanned along these epipolar lines for a matching group ofpixels. When a matching pixel set is found, the location in image B isnoted. We now know matching pixel windows and their correspondinglocations in each of image A and B. The distance between the two imagingapparatuses is known, along with the angles of projection and imagecapture for each camera/image apparatus. Triangulation methods can nowbe used to determine the 3D spatial coordinates of the objects imaged onthose pixel locations in each image. This process is continued for eachpixel group location to obtain a 3D point cloud of the imaged object(s)for each frame.

In stereo depth decoding of two images containing a reflected pattern,the features are not decoded as in the single image case. Rather, thepattern features act as a very dense and high contrast “artificialtexture”. “Windows” of pixels are compared between the two images tofind matching pixel areas. Furthermore, since feature types do notrepeat themselves along the epipolar lines when using projectiontechniques of the present embodiments, each small “window” of pixels isunique along respective epipolar lines. Again, these windows may notinclude full pattern features, but only parts of features. In eithercase, the features act as unique “artificial texture” along the epipolarline on top of the texture of the imaged objects.

Surfaces that are likely to lead to feature identification error insingle imaging systems include, for example, high textured surfaces,transparent and semitransparent surfaces and dark surfaces. This isbecause features of the encoded light projected onto high-texturedsurfaces are likely to become deformed and thus not decodable as aresult of reflectance from the high-textured surface. Although reflectedfeatures may not be decipherable, they still add unique texture alongepipolar lines to the already textured image. The comparison between twocaptured 2D images is made easier as unique “windows” can be matched toeach other. The addition of the reflected pattern aids in the stereomatching process. In non-textured or low textured surfaces, such aswalls or grass, features may be extracted from one image, as describedin previous embodiments.

The additional imaging device thus, firstly, provides “anotheropportunity” to derive the depth coordinates of a given scene of a givenimage frame. In total, 3D coordinates of a given scene using theconfiguration of the present embodiment are computed three times. Oncein each image separately through reflected feature decoding and a thirdtime from the comparison of correspondences between the first and secondimage.

The system geometric constraints that are to be imposed to enable thestereo depth decoding in addition to single image decoding are asfollows: A suitable displacement, sometimes termed baseline, is setbetween a first imaging device and the projector and between a secondimaging device and the projector. The two imaging devices are ideally onopposite sides of the projector, each at 180 degrees from the projector.Thus, the two baselines are parallel to each other and the three opticalcenters (projector and two imaging devices) are on a single line oralmost single line. The order of projectors and imaging devices mayvary. The epipolar separation techniques, however, impose the constraintthat all projectors and imaging devices be situated in a substantiallystraight line in relation to each other. The projected pattern isprojected at a suitable tilt angle, as described in above embodiments,to ensure epipolar separation both along epipolar fields between eachimaging device and the projector as well as the epipolar fields betweenthe two imaging devices.

Practically, it may further be desirable to position the projectorcloser to one of the imaging devices, say the first imaging device. Thesmaller distance between the first imaging device and the projector isparticularly suited for decoding of features reflected from closeobjects. The medium displacement between the second imaging device andthe projector is particularly suited for decoding of features reflectedfrom objects still farther away. Finally, the large displacement betweenthe first and second imaging device is suitable for stereo matching ofimaged distant objects. Since the projector has limited energy, theprojected pattern may often not reach distant objects. In such a case,the task of extracting depth is carried out through stereo matchingwithout reflected features, what is referred to as passive stereomatching. Therefore, the largest baseline is reserved for the task ofstereo matching.

Reference is now made to FIG. 25, which is a simplified flow chartshowing the steps in the process of generating a 3-D map of an urbanregion. In the example of FIG. 25, the urban region is not limited to asingle “scheme” or landscape. Instead, multiple 3D images are capturedfrom different locations and different angles, for example, by mountinga projector and/or image apparatus on a moving vehicle and capturingimages when the moving vehicle is at different locations.

The flow chart shows the example of a projector and two imagingapparatuses, however, other configurations with more projectors andimaging apparatuses are possible. The reason for use of at least twoimage apparatuses in the current embodiment is, as explained above, toenable stereoscopic depth measurement in addition to the activetriangulation methods described until now using a single imagingapparatus.

The process is as follows: First, in step s1, a two dimensional codedlight pattern is generated as in previous embodiments. This twodimensional pattern should be structured so that the epipolar separationtechniques discussed above are capable of being implemented. The patternP1 in previous embodiments is especially suited for urban modeling asthe pattern is scalable and can be made rather large for urban objects.Second, in step s2, the pattern is projected onto objects in urbanscenes from a projector(s) mounted on a moving vehicle. These urbanobjects may include buildings, cars, greenery, street signs, people, orany other object. Projection techniques are as discussed above to ensureepipolar separation.

In step s3, images are captured from each of the two imaging apparatuseswhich are located at preferably slightly different angles to the scene.The images are preferably captured simultaneously and show a combinationof the original texture and the pattern. In step s4, each of the twocaptured images is analyzed independently, as in previous embodiments,according to features and their locations along respective epipolarlines. In addition, in step s4, a comparison using stereo correspondenceas discussed above is carried out between the two images. In thiscomparison, similar pixel areas in both images are identified. Asdiscussed, similar pixel areas indicate where the same imaged locationappears in each of the images. A given pixel area in one image issearched for along certain epipolar lines in the other image and acorrespondence is found.

Now, correspondences between features in each image and the originalpattern are found out in step s5, as described in embodiments above. Inaddition, in step 5, relative locations between corresponding pixel“windows” in the two images is computed. From correspondences, 3Dspatial coordinates are found using triangulation, seen step s6.

As discussed above, the comparison between images allows for thederivation of 3D coordinates from rich texture surfaces, dark surfaces,locally unsmooth surfaces, transparent surfaces, shiny surfaces, etc,that under single pattern conditions could likely be identifiederroneously or not at all. Taken together, the three 3D coordinatemappings of the imaged objects allow for the construction of a 3D pointcloud of the imaged scene, seen as step s7. This point cloud may befurther processed to obtain a mesh or 3D surface, seen as step s8. The3D surfaces may then be further processed together with texture datafrom additional cameras or CCDs. Aerial data of the urban scene may alsobe added. Such texture and aerial data may complement the depth 3D depthmap obtained through steps s1-s8.

Steps s4 through s9 are typically carried out by an image processingdevice or devices of various kinds known in the art. Texture data may beobtained in a number of ways. One way is by adding an additional CCD.Another is by adding a tunable filter to a single “switchable” CCD, suchthat at certain time intervals the CCD captures pattern and thus depthinformation, and at other intervals the CCD captures textureinformation. Still another method is by simply adding atextured-dedicated camera to work in parallel to the depth capturingsystem.

As stated, the single pattern method employed in preferred embodimentsallows for relative motion between the imaging/projector system andimaged urban objects. Therefore, the method and apparatus areparticularly suited for an implementation where the imaging apparatus ismounted on a ground-vehicle moving at various speeds throughout an urbanenvironment.

To obtain dynamic scene modeling in three dimensions, steps s2 throughs8 are repeated and a sequence of three dimensional images obtained.Processing of data from moving objects may occur in real time throughappropriate processing software and hardware to derive surfaces andshapes from the 3D point clouds of the objects and their movement in 3Dspace over time.

As the present embodiment in FIG. 25 may be implemented throughprojection of a single pattern structured light code, the speed of imagecapture is typically only limited by the speed of the imaging apparatusin which the pattern is implemented and thus a rapid 3D modeling of citylandscape is possible. The above embodiment of urban scene modeling isnot limiting, and other embodiments utilizing the methods andapparatuses for stereo decoding are possible.

Stereo matching, either passive or active, provides a much more densesampling of the imaged scene. This is because the content of any givenpixel of a first image provides a disparity measurement in relation tothe same content appearance in a second image. This disparitymeasurement means a depth value for each given pixel. So if the CCDsensor contains X pixels, X depth coordinates are derived.

In contrast, in decoding of single images as explained above, eachfeature that is decoded has a certain spatial area on the sensor. Let'ssay typically 10 square pixels, so a decoded 2D image from which 3Dspatial coordinates are triangulated has X/10 depth coordinates. So astereo matching implementation has 10 times more depth coordinates, andthus denser sampling.

Moreover, in stereo matching, sub-pixel resolution is possible byperforming the matching process between two up-sampled versions of thetwo compared images. For example, each 4×4 block of pixels is up-sampledby a factor of 4 to a size 16×16 pixel blocks in each of two images. Acertain up-sampled block is chosen in a first image. Then a similarblock to this up-sampled block of pixels is searched for in the secondimage. The correspondence between pixels is now in terms of up-sampledpixels. So if a disparity of say 21 pixels is found between the twoimages, this is equal to 21/4=5.25 original pixels. The remainingquarter pixel is the sub-pixel resolution in terms of original imageresolution.

The method and apparatus of motion based three dimensional image captureand depth measurement of the present embodiments has particularapplication to the medical imaging field. Surface images that determinethe shape and dimensions of an anatomical surface part in threedimensions may be computed in real time for modeling of both stationaryand moving surface anatomy. The 3D point clouds, determined through theembodiments above, may be further processed to obtain surfaces andshapes of the objects and their movement in 3D space over time. Thesesurfaces and shapes are referred to as triangulation meshes.

Such 3D modeling data may be particularly useful for various medicalapplications including, but not limited to, external three dimensionalimaging for tumor analysis, skin growth analysis, bone structureanalysis, dental surgery applications, maxillofacial and otherosteotomic surgeries, skin cancer and melanoma diagnosis, prostheticapplications, rehabilitation purposes, sports medicine applications andplastic surgery and other aesthetic medical procedures.

In the fields of reconstructive, plastic and dental surgery, to name afew, biomechanical modeling may utilize dynamic 3D modeling ofanthropometric data in surgery planning and post surgery review. Theexpected aesthetic result of surgery is important to a patient and theimaging system of the current embodiments allows for soft tissuepredication in three dimensions. Post surgery review of anatomicalchanges, such as breast implants and reductions, dermal fillers, faceand neck lifts, can be carried out through use of the 3D imaging systemof the current embodiments.

For rehabilitation applications skeleton modeling of the anatomicalpart, such as discussed in U.S. Pat. No. 6,133,921, may be carried outbased on analysis of the 3D point cloud. Motion analysis of the skeletonmodel may then be performed. In the area of dental applications, 3Dmaxillary modeling and other facial bone analysis is possible. Otherapplications include measurement of tumor shape and size, reconstructivesurgery planning and diagnosis, dental and plastic surgery, prosthetics,rehabilitation and skin cancer analysis.

Reference is now made to FIG. 26, which is a simplified flow chartshowing the steps in the process of dimensional image capture of surfaceanatomy according to the present embodiments. In step 300, a twodimensional coded light pattern is generated. Next, in step 302, thegenerated pattern is projected onto an external anatomical part. As skinsurfaces are naturally low texture surfaces, the structured lightpattern of the current embodiments is naturally suited for skin surfaceimaging. In step 304, a 2D image of the anatomical part and thereflected pattern is captured by an imaging device. In step 306, thecaptured image is sent to a processor for extracting the reflectedfeature types and their locations along respective epipolar lines in thecaptured image. The locations of the features along their epipolar linesare then associated with 3D coordinates on the imaged anatomical partfrom which the features were reflected, step 308. This process ofcorrespondence between feature locations along epipolar lines and 3Dspatial coordinates determines the anatomical shape. The process iscarried out through triangulation techniques, as discussed above. Foreach identified feature in the 2D image, a corresponding 3D coordinateis thus derived indicating the point in space at which that featurereflected off of the anatomy part. Through a compilation of all such 3Dcoordinates, a 3D point cloud is derived that gives a three dimensionalmap of the imaged anatomical part(s), step 310. If the anatomical partis moving, then steps 304 through 310 are repeated and a sequence ofthree dimensional coordinates in space is obtained. This sequencecomprises a point cloud data set over time from which a skeleton inmotion may be generated, step 312.

In preferred embodiments, this stationary or dynamic skeleton model isfurther processed and preferably output to a 2D or 3D screen forviewing. As mentioned, the three dimensional coordinates may beprocessed in real time for either stationary or moving anatomy. Throughpost processing, for example, the point cloud and/or skeleton model istransformed into a 3D surface representing the dimensions of the imagedanatomical part. The 3D imaged surface, both in motion or stationary,can be used in the applications shown at the bottom of the figure anddiscussed above.

In an alternative embodiment, particularly in prosthetic orrehabilitation applications, the anatomical part may be covered withclothing or other material. Furthermore, the above embodiments may beimplemented together with additional imaging devices, in particular oneor more texture based imaging devices, to obtain textural informationtogether with the depth information. Examples of such implementationsinclude RGBZ, splats, color voxels and textured mesh. Typically, theseimplementations involve the addition of a textured-dedicated camera towork in parallel to the depth capturing system. This texture basedinformation may allow for additional diagnosis of ailments associatedwith skin color as well as increased visibility by avoiding occlusions.Moreover, such an additional texture-based camera allows for thereconstruction of high-contrast, high-textured, non-smooth surfaces suchas hair.

Furthermore, other embodiments may preferably utilize a second imagingdevice as discussed above. Stereo active triangulation techniques offerseveral advantages. Less occlusions occur in the obtained 3D image, asthe object or anatomical element is now imaged from more than oneviewpoint. For high-contrast, high textured surfaces mentioned above,such as body parts covered by hair, freckles, melanoma, stereo imagingis preferred as discussed above. Finally, higher resolution is obtainedwith stereo matching as discussed above as well. As the human body isgenerally a low textured surface, the single image decoding and 3Dspatial coordinate derivation is utilized for most applicationsrequiring geometric shape of stationary or moving body parts.

Many medical applications require 3D motion capture for anatomy motionanalysis. Furthermore, the human body is not static even at rest. As aresult, dense sampling is necessary for each frame.

Still further applications of motion based three dimensional imagecapture and depth measurement of the present embodiments are in theautomotive arena. One important application is a 3D backup camera formotor vehicles. The ability to measure depth of potential obstacles inthe path of a vehicle in reverse and warn the driver in time is acritical safety need. For any given identified object in the path of areversing vehicle, it is desired to determine the classification of theobject such as size, precise location, object-vehicle relative velocityand, importantly, whether the object is animate or inanimate.

A typical flow chart for the automotive application is shown in FIG. 27.As mentioned above, at least one projector and at least one sensorapparatus are installed on the rear side of the vehicle. The projectorprojects the 2D coded light pattern, step 324, possibly using IR lightfrom a laser LED projector. The pattern is reflected from bothstationary and moving objects in a defined lateral and depth rangebehind the vehicle. The sensor then captures the reflected pattern andtexture information, seen as step 326. This data is then processed, step328, by an onboard processing unit to identify features and theirlocations in 2D image along respective epipolar lines. These locationsof features are then corresponded to distances of identified objects,step 332. These distances are then used to warn driver either audibly orvisually, on a display screen on the dashboard for instance. Thisprocess is repeated continuously over time to provide continuous realtime information to the driver.

Preferred embodiments utilize narrowband monochromatic imaging which issimpler and cheaper than capturing color images. Monochromatic imagingallows the projection of low cost invisible IR light.

It is expected that during the life of this patent many relevant devicesand systems will be developed and the scope of the terms herein isintended to include all such new technologies a priori.

The terms “light pattern”, “light code”, and “light code pattern”, areused herein to refer to any light encoding technique, including, but notlimited to, structured light, coded light and other equivalents known inthe art.

The term “epipolar co-linear” simply means that two independent featuresappear on the same epipolar line in a given 2D image.

The term “reflection point distance:image point location relation”refers to a one to one relationship between each point in the captured2D image and a point in 3D space. This relationship can be provided in,for instance, non-volatile memory such as flash memory.

The term “feature-type-inhomogeneous plurality” refers to a set offeatures wherein each feature is unique.

The object(s) imaged by the method and apparatus described herein can bealternatively referred to as the “target scene” or “calibration scene”in the case of calibration.

In addition to other image capture apparatuses, a 2D array ofphotodetectors may be used as well.

The term “insensitive to epipolar line ordering” is equivalent to saying“insensitive to scene distance”.

It is appreciated that certain features of the invention, which are, forclarity, described in the context of separate embodiments, may also beprovided in combination in a single embodiment. Conversely, variousfeatures of the invention, which are, for brevity, described in thecontext of a single embodiment, may also be provided separately or inany suitable subcombination.

Although the invention has been described in conjunction with specificembodiments thereof, it is evident that many alternatives, modificationsand variations will be apparent to those skilled in the art.Accordingly, it is intended to embrace all such alternatives,modifications and variations that fall within the spirit and broad scopeof the appended claims. All publications, patents and patentapplications mentioned in this specification are herein incorporated intheir entirety by reference into the specification, to the same extentas if each individual publication, patent or patent application wasspecifically and individually indicated to be incorporated herein byreference. In addition, citation or identification of any reference inthis application shall not be construed as an admission that suchreference is available as prior art to the present invention.

What is claimed is:
 1. A device, comprising: a memory configured tostore a digital image of a reflected portion of a projected structuredlight pattern that is comprised of a repeating structure of a pluralityof unique feature types; a processor configured to identify and locatefeature types appearing in the image and using known locations of eachone of a plurality of distinguishable epipolar lines on a sensor's planeand a constraint that any feature type appears at most once along anyone of the plurality of distinguishable epipolar lines, wherein adistance between the distinguishable epipolar lines is less than YC,where C is a size of a feature type in the image in the shortestdimension of the repeating structure, and Y is a number of feature typesin the shortest dimension of the repeating structure; and the processoris configured to extract depth information based on locations of thefeature types along respective distinguishable epipolar lines.
 2. Thedevice according to claim 1, wherein the repeating structure of theplurality of feature types embodies a coding matrix.
 3. The deviceaccording to claim 1, wherein the repeating structure of the pluralityof feature types is a tile.
 4. The device according to claim 1 whereinthe repeating structure of the plurality of feature types is a periodicrepeating structure.
 5. The device according to claim 1, wherein eachone of the plurality of unique feature types is characterized by aunique combination of spatial formations.
 6. The device according toclaim 1, further comprising: a projector configured to project a codedpattern, such that when the projector projects light, the structuredlight pattern is projected; and a sensor configured to capture thedigital image of the reflected portion of the projected structured lightpattern, wherein the locations of the distinguishable epipolar lines onthe sensor's plane are associated with a spatial relation between theprojector and the sensor, and wherein said projector is orientated withrespect to the sensor such that when the projector projects, theprojected structured light pattern is at an angle relative to anepipolar field associated with the spatial relation between theprojector and the sensor, and wherein the distance between thedistinguishable epipolar lines is associated with an orientation of theprojected structured light pattern relative to the epipolar field. 7.The device according to claim 6, wherein the projector is configured,such that when the projector projects the structured light pattern, anorientation of the structured light pattern with respect to the epipolarfield is limited by a limiting epipolar separation factor.
 8. The deviceaccording to claim 6, wherein the projector is configured such that whenthe projector projects, the projected structured light pattern is at anangle θ relative to an epipolar field, where θ=Sin⁻¹(P/YC), P denotes adistance by which distinguishable epipolar lines are separated, C is asize in the shortest dimension of the repeating structure of a featuretype in the image, and Y is a number of feature types in the shortestdimension of the repeating structure.
 9. The device according to claim1, wherein the device has a predefined operating range, and in theimage, two or more repetitions of a feature type appear on adistinguishable epipolar line, but within any given section of anepipolar line which corresponds to the operating range of the device,any feature type appears at most once.
 10. The device according to claim9, wherein the processor is configured to extract depth informationbased on a location in a given section of a distinguishable epipolarline of a given feature type on the distinguishable epipolar line. 11.The device according to claim 1, wherein Y equals
 1. 12. The deviceaccording to claim 1, further comprising: a sensor configured to capturethe digital image of the reflected portion of the projected structuredlight pattern; and a projector, the projector comprising: a patterningelement; and a light source, and wherein the patterning element isconfigured, such that when the light source illuminates the patterningelement, the patterning element manipulates light from the light sourceto provide the projected structured light pattern, and wherein thepatterning element is orientated with respect to the sensor such thatwhen the projector projects the structured light pattern, the projectedstructured light pattern is at an angle relative to an epipolar fieldassociated with the spatial relation between the projector and thesensor.
 13. A method, comprising: obtaining a digital image of areflected portion of a projected structured light pattern that iscomprised of a repeating structure of a plurality of unique featuretypes; identifying and locating feature types appearing in the imageusing known locations of each one of a plurality of distinguishableepipolar lines on a sensor's plane and a constraint that any featuretype appears at most once along any one of the plurality ofdistinguishable epipolar lines, wherein a distance between thedistinguishable epipolar lines is less than YC, where C is a size of afeature type in the image in the shortest dimension of the repeatingstructure, and Y is a number of feature types in the shortest dimensionof the repeating structure; and extracting depth information based onlocations of the feature types along respective distinguishable epipolarlines.
 14. The method according to claim 13, wherein the repeatingstructure of the plurality of feature types embodies a coding matrix.15. The method according to claim 13, wherein the repeating structure ofthe plurality of feature types is a tile.
 16. The method according toclaim 13, wherein the repeating structure of the plurality of featuretypes is a periodic repeating structure.
 17. The method according toclaim 13, wherein each one of the plurality of unique feature types ischaracterized by a unique combination of spatial formations.
 18. Themethod according to claim 13, further comprising projecting thestructured light pattern such that the projected structured lightpattern is orientated, at an angle relative to an epipolar fieldassociated with a spatial relation between a projector that is used toproject the structured light pattern and a sensor that is used tocapture the image, and wherein the distance between the distinguishableepipolar lines is associated with the orientation of the projectedstructured light pattern relative to the epipolar field.
 19. The methodaccording to claim 18, wherein the orientation of the projectedstructured light pattern with respect to the epipolar field is limitedby a limiting epipolar separation factor.
 20. The method according toclaim 13, wherein in said image two or more repetitions of a featuretype appear on a distinguishable epipolar line, but within any givensection of an epipolar line which corresponds to an operating range of adepth sensing device that is used to provide the projected structuredlight pattern and to capture the digital image, any feature type appearsat most once.
 21. The method according to claim 20, wherein saidextracting depth information comprises, extracting depth informationbased on a location in a given section of a distinguishable epipolarline of a given feature type on the distinguishable epipolar line. 22.The method according to claim 13, wherein Y equals
 1. 23. The methodaccording to claim 13, where said projecting comprises, projecting thestructured light pattern at an angle θ relative to an epipolar field,where θ=Sin⁻¹(P/YC), P denotes a distance by which distinguishableepipolar lines are separated, C is a size in the shortest dimension ofthe repeating structure of a feature type in the image, and Y is anumber of feature types in the shortest dimension of the repeatingstructure.
 24. A non-transitory machine-readable storage mediumincluding instructions which, when performed by one or more processors,causes the one or more processors to perform a method of extractingdepth information from a scene, the method comprising: obtaining adigital image of a reflected portion of a projected structured lightpattern that is comprised of a repeating structure of a plurality ofunique feature types; identifying and locating feature types appearingin the image using known locations of each one of a plurality ofdistinguishable epipolar lines on a sensor's plane and a constraint thatany feature type appears at most once along any one of the plurality ofdistinguishable epipolar lines, wherein a distance between thedistinguishable epipolar lines is less than YC, where C is a size of afeature type in the image in the shortest dimension of the repeatingstructure, and Y is a number of feature types in the shortest dimensionof the repeating structure; and extracting depth information based onlocations of the feature types along respective distinguishable epipolarlines.
 25. The method according to claim 24, wherein the repeatingstructure of the plurality of feature types embodies a coding matrix.26. The method according to claim 24, wherein the repeating structure ofthe plurality of feature types is a tile.
 27. The method according toclaim 24, wherein each one of the plurality of unique feature types ischaracterized by a unique combination of spatial formations.
 28. Themethod according to claim 24, wherein in said image, two or morerepetitions of a feature type appear on a distinguishable epipolar line,but within any given section of an epipolar line which corresponds to anoperating range of a depth sensing device that is used to provide theprojected structured light pattern and to capture the digital image, anyfeature type appears at most once.